Talk:Gem Store/data

New storage

Currently, the gem store items are stored on the following 11 pages:

• Gem Store/Style
• Gem Store/Utility
• Gem Store/Toys
• Gem Store/Upgrades
• Gem Store/Combos
• Gem Store/storage/Style
• Gem Store/storage/Utility
• Gem Store/storage/Toys
• Gem Store/storage/Upgrades
• Gem Store/storage/Combos
• Gem Store/historical Gem Store/storage/historical

requiring the wiki editors to move (copy-paste) the {{gem store table row}} templates after each gem store update, resulting in an extremely tedious work.

Here, I'm trying to put all information on one page, stored silently to avoid any unnecessary computation, using the new template {{gem store entry}}. Comparing the two templates:

{{gem store table row
| item =
| cost =
| amt =
}}

{{gem store entry
| item = 
| cost = 
| qty = 
| section = 
| subsection = 
| status = 
}}

The new template has more parameter that needs to be set each time, namely section and status, that previously was handled by the page itself, but now since everything is on the same page and sorted alphabecially is has to be set every time. However, once set, the section doesn't change, furthermore, it allows multiple sections Edit: not true, in order to sort the items properly only one section is allowed, furthermore we need also the parameter subsection, e.g. subsections "Outfit", "Glider", etc... in the section "Style", to refine the sort even more.

The real benefit of this new template should be the parameter status allowing the following parameters: available, unavailable (for currently unavailable items) and historical (for permanently unavailable items). The idea is that simply switching from "available" to "unavailable" is faster than coping the whole {{gem store table row}} template in the correct page.

In order to demonstrate, I already added the template (note that the new template does NOT set any properties yet) and dropped all gem store items into this page.

The gem store tables would then be created with smw queries on the according pages, e.g. available on the gem store page, unavailable and historical on subpages.

Any thoughts? It would be nice to get some feedback espcially from active gem store editors (or maybe not-so-active editors how got frustrated by the old scheme), e.g. is this an actual improvement or not, what else could be improved? --Tolkyria (talk) 20:26, 5 February 2021 (UTC)

I hope can let status auto switch for available and unavailable by end date. otherwise that mean still need edit two time when available and unavailable.

Also if known item is historical (permanently unavailable items) that actually can move to a subpage or keep your way is fine. Wong8888 (talk) 21:48, 5 February 2021 (UTC)

Adding dates isn't really an option right now: first, it would somehow require backwards compatibility from the last eight years (using the {{Gem Store history table row}} data gives ~700 unknown dates which somehow break the whole approach), and second, it would require an easy way to add dates for any future update as introducting a date-based approach is kinda definitive and can't be reverted after once implemented (and adding dates by hand is definitely more work than using browser search and adding or deleting un from available). However, I think given this more universal approach, probably adding dates later would much easier than modifiy the old {{gem store table row}} templates spread on all the subpages. Therefore, implement this unified approach first, if the community approves it, and then see how we could further improve it.

You are right, permanent historical items could be on a subpage, e.g. /data (historical). --Tolkyria (talk) 22:25, 5 February 2021 (UTC)

I really like your work so far, implementing it would be an immense help to anyone who is interested in maintaining Gem Store pages without the need to edit 10 different subpages. I am definitely for implementing it. ~Sime 16:25, 6 February 2021 (UTC)

Though dates would be awesome I do see why it could be an issue. One thing I am wondering about is if a property for sales could be added. So you'd have the base `price` property, and then maybe `discounted price` when there is a sale. I imagine if none is set default to main price. This would though bring the issue of items that have a discount for the first purchase, and then full price after; which I'm not quite sure how it would be addressed.

I do think though that this will be a great addition and make keeping things up to date far easier (because let's be honest editing those current pages suuuuuuucks) -Darqam 19:44, 6 February 2021 (UTC)

For me the discounted price sounds more like a gem store history table feature, and then we are back to the date-approach, e.g. with input <start date>;<end date>;<discount percentage> we could compute the discounted price for a certain time frame easily (or the other way around, discounted price computes the discount percentage). But without any way to somehow get this dates automatically, I fear that it would become even more tedious to edit the gem store than it is now, i.e. exactly the opposite from my intention to simplify it. So right now I can't see any reasonable way to do it, however, once everything would have settled, that's definitely something we can think about. --Tolkyria (talk) 20:37, 6 February 2021 (UTC)

Yes please, I did style earlier this week and I'm still 90% sure I missed at least 5 items on the currently unavailable page that I was supposed to transfer. I like what this does and I think it would save us a headache or six. Would it be possible to have however the table shows up to be organizable by availability/section with this? I am not familiar with templates. Nightwhisper (talk) 00:27, 7 February 2021 (UTC)

So, you suggest to

1. sort by section and subsection rather than sorting alphabetically by item name.
2. also have a visible output to somehow see if the added content is correct.

Okay. --Tolkyria (talk) 11:34, 7 February 2021 (UTC)

Gut feel would be it'll be easier to update if it's just A-Z and the subsection is parameterised. I think (if any) the visual output should be extremely basic, possibly just a wiki link to the item. -Chieftain Alex 11:56, 7 February 2021 (UTC)

With this feedback, I'm questioning my approach, maybe just A-Z is too much (with some plain wikitable, no icons), I don't know. Maybe we should keep the five sections /Style, /Utility, /Toys, /Upgrades, /Combos and only merge the /storage into those while keeping the current template gem store table row together with introducing an availability parameter, i.e. a reduction from 10 to 5 subpages. Sure for an automatized approach in the future one page would be better; but as I don't have internet access next week, it might get a little bit too much for now. --Tolkyria (talk) 12:55, 7 February 2021 (UTC)

Okay, sometimes I'm questioning my ideas maybe too much. Although the template gem store table row is more natural to edit, I think having only one page (/data) instead of five (/Style, /Utility, /Toys, /Upgrades, /Combos) is the better approach, as it is more flexible and supports potential upcoming improvements.

I have added a visible summary of the used paramters to the template gem store entry, see Gem Store/data and Gem Store/data (historical). --Tolkyria (talk) 16:11, 7 February 2021 (UTC)

There's something to be said for how elegant the A-Z list is, I love how simple it appears to be on the outside. -Chieftain Alex 18:53, 7 February 2021 (UTC)

This way of updating the gem store would definitely make the physical part of editing it much easier so i support this.

I also support the idea of the historical items being on their own page; as long as there's a link to the other page on both pages so they're easily found.

As to what could be improved; it would be great if the /v2/gemstore/catalog endpoint already listed in /v2.json could finally come to life, as that seems like it could be what we would need here. Other than that i can't think of anything for the time being.

With regards to computing the discounted price vs the discount percentage; provided it gets implemented, i'm for computing the percentage.

Oh and i feel like A-Z would be easier to navigate; though a note about ctrl+f couldn't hurt. Nightsky (talk) 23:48, 11 February 2021 (UTC)

Transition protocol

If we are going to use this approach, this are in my opinion the required steps to provide a smooth transition between deactivating the template gem store table row and enabling the template gem store entry:

1. 1 Create the template gem store entry but do not set properties yet.
2. 1 Create the property Has vendor subsection to allow even further categorization, e.g. Style (section) -> Outfit (subsection) or Style (section) -> Glider (subsection), used for smw sort = Has vendor section, Has vendor subsection, Sells item
3. 1 Define the variable subsection in all gem store subpages with {{#vardefineecho:subsection|<subsection>}} and set the property Has vendor subsection to {{#var: subsection}} in the currently used template gem store table row.
4. 1 Create the pages Gem Store/data and Gem Store/data (historical) by using smw queries to get the currently set values (including the recently set subsections). Proper intro box is missing though!
5. 1 Create the template gem store table, using the result format gem store table result format, taking the parameters vendor (default is pagename: "Gem Store", "Gem Store/unavailable" and "Gem Store/historical") and section ("Style", "Utility", "Toys", "Upgrades", "Combos")
6. 1 Create the text property Has vendor anchor which is set to the subobject name gemstore{{#var:gemcnt}} in order to use it as an anchor in the result format gem store table result format. Note that simply using the suboject name would link to the /data subpage which doesn't look very appealing to wiki users.
7. 1 Modifiy the templates vendor table and vendor list to use Has vendor anchor if it is set, otherwise use the subobject link: {{#if: {{{Has vendor anchor|}}} | [[{{{Has vendor|}}}#{{{Has vendor anchor|}}}|...]] | [[{{{1|}}}|...]] }}. This links to a table row of the smw template gem store table which has id="gemstore{{{Has vendor anchor|}}}" (see also: 10.).
8. 0 Deactivate the subobject declaration in the template gem store table row.
9. 0 Activate the subobject declaration in the template gem store entry.
10. 0 Add the template gem store entry gem store table to the three vendor pages: "Gem Store", "Gem Store/unavailable" and "Gem Store/historical" (move the original page to somethink like "Gem Store/storage/historical" to keep the page version history). (edited later by Alex for clarify to refer to Template:Gem store table, not Template:Gem store entry)
11. 0 Archive and mark the gem store subpages using gem store table row as historical to preserve the page version history (do not delete these pages as the history provides valuable information).

--Tolkyria (talk) 20:37, 6 February 2021 (UTC)

Sorry, I lost the momentum, also seeing e.g. Hammer of the Three Realms Skin neither getting added to the old or the new scheme, or Armistice Bastion Pass getting only removed from Gem Store/storage/Utility but not added elsewhere, etc... is not really motivating either. --Tolkyria (talk) 15:35, 17 February 2021 (UTC)

Given that you've done the hard bit, we should definitely be finishing this. As soon as we kill off the other subpages there will be no choice but to update this page! :D

I had a question with respect to available+unavailable vs historical. It's a bit tricky to tell the difference between these isn't it? Will we be anywhere near the subobject limit (if there is one) if we kept everything (available+unavailable+historical) on the same page? -Chieftain Alex 21:26, 17 February 2021 (UTC)

The parameter availability=historical is only used on Gem Store/data (historical) (was proposed by some editors, see above, as these most likely never come back), only availability=available/unavailable are stored on Gem Store/data. That's also bascially the current idea: Gem Store/storage sets the canonical name "Gem Store (currently unavailable)" and Gem Store/historical sets the canonical name "Gem Store (historical)". However, with this page split we could set the historical automatically (e.g. with some #var) or add the parameter historical=y instead, both would reduce it to the two options availability=available/unavailable.

I saved all of this subobject once on my sandbox page (including the 169 historical ones; didn't set vendor section to avoid mainspace appearance which is used in smw sort) and it was totally fine (over 1000 subobjects). So we have some space for new gem store items, assuming that we do not really increase the template complexity. Otherwise we could always split it two pages: A-M and N-Z. --Tolkyria (talk) 21:52, 17 February 2021 (UTC)

I compared all the wiki pages on Gem Store/data and Gem Store/data (historical) with the output of "Widget:Gem Store 2" - almost every item but 11 still exists in the gemstore catalogue. I suggest those 11 items are totally historical, but everything else could technically return. Items:

Character Slot and Experience Scroll Package
Mini Egg
Black Feather Wings Backpack Set
Consortium Swim Speed Boost
Consortium Swim Speed Boost
Limited-Use Gift Finisher
Limited-Use Gift Finisher
Limited-Use Snowman Finisher
Limited-Use Snowman Finisher
Mystic Key
White Wings Backpack Set

-Chieftain Alex 20:20, 18 February 2021 (UTC)

Well, theoretically yes (based on what you found there), but practically I would say no. Here I trust the wiki users more that put them on the /historical page than the gem store catalogue. Taking a look at Gem Store/historical, my source of Gem Store/data (historical):

• Wintersday weapons: Sime quoted on the version history of the weapons: "Was available only once, 99,99% it is not gonna return to Gem Store."
• Town clothing: these won't return.
• Unidentified Dye: sold for a few silver in trading post, so unlikely.
• Living World Season 1 stuff, Instant Trait Reset, old Transmustion stuff, old Boosters, etc..., very unlikely that those will return.
• Not sure about the finishers, but those are sold by Black Lion Statuette, so most likely they won't return either.

Overall, I can't really see any item that should be moved right now. Again, I trust the wiki editors and would leave the historical data as it is. If necessary, we can always move few items rather than moving now almost all that in the end will never return.

Also, there clear difference between "currently unavailable", i.e. may return in the future, and "historical", i.e. most likely may not return in the future (unless some super rare case kicks in), which I think should be still emphasised. This means for the wiki users that the could either wait for their wanted item to return or that they clearly see that it won't return in gem store at all. --Tolkyria (talk) 22:19, 18 February 2021 (UTC)

I can generate some pretty close wikicode per User:Chieftain Alex/sandbox#Listings but I'm currently trying to think how we can convert gem store item names to wiki page names.

Maybe if I were to annotate all gem store item pages with the guid I could do a SMW request for gemstore items. -Chieftain Alex 15:08, 20 February 2021 (UTC)

Just realised dataId is the item id... -Chieftain Alex 15:23, 20 February 2021 (UTC)

Wow, very nice! Side remark: we need also a subsection to properly group it within one section, simply sorting alphabetically wouldn't be userfriendly at all. It should be basically the gem store catalogue sections, maybe grouping the obvious ones, e.g. Medium Armor, Light Armor, Leggings, etc... is subcategory Armor, Mining Tools, Logging Tools, Harvesting Tools is subcategory Gathering tools, etc...

The idea of this whole redesign process is, the new template and your gem store widget, to make it as simple as possible and to edit as less pages as needed (preference is only one page, namely /data). Having to set the guid on every page in order to make it work properly ruins the complete idea. That's another step the wiki editors have to perform when creating new items, in my opinion a tedious (and hopefully avoidable) extra step. The dataId works for the suffix (container) but not for the (package), as those contain more than one.

However, I think the suffix are pretty straight forward, they are either (consumable), (container), (package) (and one times (item)). So what about checking first if the page exists with one of the suffixs, and if not take the item name? Sure, that's kinda hacky, but it's the only useable solution that comes into my mind right now. --Tolkyria (talk) 15:46, 20 February 2021 (UTC)

Edit: Or some mixture: use the dataId (i.e. the item id) if it's set and for entries without dataId but with contents check if the wiki page exists with the suffix (package), if yes then add the suffix to the item name. --Tolkyria (talk) 15:52, 20 February 2021 (UTC)

Ok I've got the wiki names into the gem store widget, using stuff we devised for the filter table widget.

I'm up for sorting the data differently, I just need to know what we want. I'll put the item to categories (note: plural) data in Guild Wars 2 Wiki:Sandbox so you can see what I'm playing with. At the moment to figure out the "category" I've got a functin named function wikiSection which converts the array of category names into the most likely main category. For subcategories I suppose I can do the same, perhaps with the subcategories the other way around.

At the moment I've ignored historical items (they get filtered out right at the start). I don't know if that's a problem or not. -Chieftain Alex 16:14, 20 February 2021 (UTC)

I have added the in-game gem store window sections and their subsections to the sandbox. What about using exactly those? No idea why we have five sections, i.e. where our /Combos comes from. Note that the template requires a section (Style, Utility, Toys, Upgrades) but it do not necessarily require a subsection, i.e. it will use "Uncategorized" on default. --Tolkyria (talk) 16:28, 20 February 2021 (UTC)

Yeah combos was weird. If we stick with the 4 root categories it seems easier. I've updated the output in my sandbox. I think this is pretty much where it needs to be for the available content. I ended up sorting alphabetically by Section then by Name - seems too contrived to sort by Section > Subsection > Name imo.

Next puzzle is what to do with non-current content. Crazy theorycraft time: Theoretically I suppose I could (1) get the previous raw wikitext content from both "Gem Store/data" + "Gem Store/not available" pages, removing all available/unavailable lines, (2) combine them and (3) do a diff with the main output of the widget minus the available lines. (4) Anything which is in both would get removed, and (5) the rest would form the wikitext for the not available page. The widget should then either provide the new content of both pages either for copying, or write the edit directly. -Chieftain Alex 17:57, 20 February 2021 (UTC)

Sounds good, except the point that I mentioned yesterday. So many items listed on Gem Store/historical are clearly historical but still appears in the gem store catalogue, e.g. Town Cloths, Transmutation Crystal, etc... Reconsidering these items as currently not available instead of correctly marking them as historical is definitely a step back in our wiki documentation and should be avoided. Those items should be clearly marked as historical, maybe you could add black list of historical items that are excluded in the widget output, those are stored on /historical and are left untouched. For example you could automatically generate this black list with smw, all items that are stored on /data (historical) aren't taken into account in the output. --Tolkyria (talk) 18:25, 20 February 2021 (UTC)

Looking pretty fly. In retrospect I think the dividing headers probably help with reading the page as a non-robot so I'll see if I can modify the script a bit (both to initially remove the header, and then to re-add them). -Chieftain Alex 21:13, 24 February 2021 (UTC)

Indeed, the widget is really amazing, I really like that you are using /data to get the unvailable ones.

But then we should only sort alphabectially by name and not first by section and then alphabetically by name afterwards. We could shift the header into the template by storing the last initial letter and compare it with the current one of the item, but this can't be used for section edits, also a little bit too expensive doing ~1000 comparisons for 26 letters (plus some numbers). --Tolkyria (talk) 21:28, 24 February 2021 (UTC)

Both the data page and the widget look excellent imo (well the widget is functionally excellent even if its a disaster for programmers).

I'm sure we can get the data sorted however we want it, and if we want dividers in its not too difficult, it would be a case of modifying the section below // Rebuild wikitext + a pattern replace exercise in function fetchOldWikitext(). I've just this minute changed it back to alphabetical sort without sections. -Chieftain Alex 21:38, 24 February 2021 (UTC)

(Reset indent) Should I clean up the mess I created so far (step 1-7)? --Tolkyria (talk) 09:08, 21 March 2021 (UTC)

Are we abandoning this project? -Chieftain Alex 16:53, 21 March 2021 (UTC)

I don't know, that's why I'm asking. The idea was to simplify the overall progress and to somehow compete with reddit or the forum listing new and returning gem store items, but here we are, around one month without any noteable progress. The current gem store with all its subpages is complicated enough, so I want to avoid that a leftover approach makes it even more complicated, thus my question: are we? --Tolkyria (talk) 18:53, 21 March 2021 (UTC)

Everyone needs a break sometimes, and it would be really a shame if this got abandoned now since it seems to be almost done based on the list? Imho I think it should be finished since it will be a great boon to everyone who works on Gem Store pages, but of course I cannot force anyone to do stuff. I would 100% help if I could but I am afraid I would break something. ~Sime 20:01, 6 April 2021 (UTC)

I forgot about this widget until you linked it on reddit. I tried to use it today and found it was broken so i've fixed yet-another-cornercase. I think the format is pretty human friendly on the data page too. Can I suggest we do continue and move the data over to use this page. -Chieftain Alex 20:44, 23 April 2021 (UTC)

Subobjects switched over, pages marked with a notice, and main page switched over to using {{Gem store table}}.

I've moved the old Gem Store/historical page + created a new one, and also created Gem Store/unavailable. Think that's it for implementation? -Chieftain Alex 17:55, 4 May 2021 (UTC)

Deduplicate duplicates

So why not list both the unavailable discounted price and the full price? -Chieftain Alex 20:55, 6 December 2021 (UTC)

Possibly we could annotate the discounted entries with a new parameter "discount = y", and unavailable "discount = y" rows would be removed on update. -Chieftain Alex 21:01, 6 December 2021 (UTC)

Sure, if you want to adjust the gem store magic widget, go for it!

Regarding the query template sold by: in order to avoid confusion about "duplicate" entries, we could reuse the already existing Has requirement-infrastructure which automatically adds an additional Notes column which displays the Has requirement property text (also the discount is actually a requirement... for a lower price, so it's kinda justified to reuse this property). For example setting "discount = y" as template parameter on this page will result in setting the SMW text property Has requirement to "Discounted price" (or whatever). This would properly explain the "duplicate" entries in the item acquisition template sold by. --Tolkyria (talk) 21:37, 6 December 2021 (UTC)

In general i have nothing against listing both the discounted and full price. Other than it seeming like a horror to me to update. But there seem to be some inconsistencies in the data produced by the widget that should be ironed out first. Specifically, at least for offers of multiple things in one purchase not currently available for a discount, if there are duplicates, then it seems that the duplicates show incorrect (increased over normal) prices for the offers marked available, whilst the correct prices are in the offers marked as unavailable; except for where the quantity is one. I've removed those so far using deduplication simply as i couldn't quite make out the exact circumstances when things where switched. And i didn't want to maintain the discounts eiter. Nightsky (talk) 23:46, 6 December 2021 (UTC)

I've modified the template code but it needs a bit more work, and prior to applying the first widget-dataset I need to manually annotate discounted items. -Chieftain Alex 22:40, 7 December 2021 (UTC)

Something weird going on (reviewed A-I and gave up)

Aurene Dye Kit has a permanent discount not shown in the UI? - 125/500/2500 ingame, versus 125/500/3125 in file
Black Lion Chest Key somehow has a 125/450/2100 gem cost ingame, but 125/625/3125 in the localstorage file
Dye Pack - same
Experience Booster - same
Heroic Booster - same
Item Booster - same
Distant Lands Mount Adoption License - ditto
Exotic Breeds Mount Adoption License - same

There's a fair number of items with hidden discounts ingame apparently. Looking at the Flame Dye Kit specifically:

• Local storage file: Price = 125 Gems; Qty = 5, Price = 625 Gems; Qty = 25, Price = 3125 Gems
• Ingame: Price = 125 Gems; Qty = 5, Price = 500 Gems; Qty = 25, Price = 2500 Gems

undeclared 20% discount?

-Chieftain Alex 22:31, 10 December 2021 (UTC)

Yes, all items that you can buy in bulk are discounted the more of them you buy. ~Sime 22:43, 10 December 2021 (UTC)

Nah, I found the answer. Some items are secretly discounted, where the "discount" flag is set to 0 (false) in the localstorage file, but "discount_pct" is set to 80% instead of 100%. -Chieftain Alex 00:21, 11 December 2021 (UTC)

With any luck that should be it working finally. -Chieftain Alex 01:14, 11 December 2021 (UTC)