[NovemLinguae]

Related #135
Related #151

[NovemLinguae]

cc @FaviFake in case you aren't subscribed yet.

Sorry for the delay on this. It's a pretty hard patch, and I've been pretty busy. Will keep trying to plug away at this every few days.

The code that does the wikicode transformation is complete. Now I need to wire it into code that generates a list of pages to do the transformation on.

I may end up doing this as two patches. It's good to split up complex patches into bite size chunks. Patch 1 would be just running the wikicode transformation on one type of page (source page? target page?), then patch 2 would generate a complete listOfPages according to the specification given and loop through it.

[FaviFake]

No! Don't be sorry! It's unfortunate that the wiki half-unknowingly forced the sole active maintainer of a volunteer project to do such a hard task all on his own. Thanks for all the effort you've put into this huge project!

I know precisely nothing about coding, but it sounds like you already did the hardest and most important part! For Patch 1, even if you check just one page it would be a big improvement. In the interim, you should run the the transformation on the Target page for discussions closed as merge or redirect.

I guess you can add the check for whether it's a redirect or it exists in patch 2, together with the rest of the list of pages to check. This is great, I'm starting to see the finish line! Thanks so much!

[NovemLinguae]

@FaviFake Sorry this one is taking so long. It is quite complex, and XFDcloser is not fun to work on.

I took a fresh look at this one tonight, and I think I'd like to add the intermediate step of making this a user script and asking you to test it out a bit. Would you mind?

Phase 1 - Please install https://en.wikipedia.org/wiki/User:Novem_Linguae/Scripts/RemoveMergeTemplates.js , then please visit a page you'd like to process. Go into the More menu on Vector 2010 (or it may be in a different spot on other skins), click "Run RemoveMergeTemplates", type in the sourcePage and nominationName, then double check the diff and make sure there's no bugs.

You can report any bugs in this ticket. A good way to report bugs might be to submit the buggy diff, then fix it in the next diff, then link the diff here.

Phase 2 - Once I am confident the wikitext algorithm works well, then I can work on an algorithm that generates the list of pages to check. I can add that to the user script, along with links next to each page you can click that would load these diffs, and then we can double check that algorithm.

Phase 3 - Finally, we'll eventually integrate all this into XFDcloser, once we're confident the algorithms work well.

As an added bonus, making this a user script gives you a tool to fix these faster until we finally get everything integrated into XFDcloser.

That all sound OK?

[NovemLinguae]

@FaviFake How's testing going?

[FaviFake]

I'm so sorry to respond 3 days later but unfortunately I've been swamped with work on another non-WMF wiki and have almost entirely ignored enwiki. I've installed the script and as soon as I open my watchlist I'll test it out, thank you so much!

[FaviFake]

Oh wow we both independently commented at the same exact time! But yeah, it's been a surprisingly busy week for me. Unless my workload increases rather than diminish, I will have finished testing next week! I can't wait for the XFDcloser updates to be over!

[NovemLinguae]

After this patch, I'm going to bail on XFDcloser patch writing and switch back to reviewing I think. Finishing that entire ticket may never happen or may take a very long time since someone will need to write the patches and I'll need to review them.

You told me that this piece of the ticket is the most important though, so I am trying to get this one to the finish line for you.

[FaviFake]

Yes this sounds perfect. The other items in my huge ticket are all really minor issues that pale in comparison to this one.

If the folks at MFD will ever decide to allow merge proposals, they are free to figure out a simpler system. This transition has been incredibly hard on everyone involved with the BTS stuff and I can't imagine what it's like to be the only one willing to write the code, not because you enjoy doing it but for the benefit of the project. Thanks for always being so helpful Novem!!

[NovemLinguae]

BTS?

[primefac]

Behind The Scenes

[FaviFake]

Sorry, Wikipedia has made me addicted to WP:3LA (three-letter acronyms)

[FaviFake]

@NovemLinguae Alright, I tested it a bit and it works great! Most of these pages were modified by me beforehand to test some edge cases, otherwise they would've all looked the same because they're placed by Twinkle and nobody adds them by hand anymore.

Successful tests where the script worked and the templates were removed correctly:

• test and test—standard test case
• test—handles redirects without "1=" parameter.
• test—processed edit with no date or discussion.
• test—handles redirects with multiple source pages, missing nomination name, and missing date.
• test—removes comments around standard dated AfD template.
• test—removes comments around standard dated AfD template when providing only nomination name to the script, where the line containing the first comment does not start with "<!--".
• test—processes multiple source templates and a redirect of {{Merge from}}.
• test—handles providing only nomination name after supplying both source page and nomination name to the script.
• test—handles redirect where date parameter equals nomination name.
• test—correctly removes template containing nomination name but not as standalone parameter
• test—correctly removes template containing source page, but not as standalone parameter
• test—successfully removes template despite the template name being all lowercase.
• test—successfully removes template when source page name matches target page name.
• test and test—removed more than one template + AfD hidden comments
• test—correct, didn't detect any eligible merge templates and do didn't run.
• test—successfully removed just one template and left the others intact

Unsuccessful tests where the script said "No changes needed!":

• test, test, and test—failed to remove template, likely due to spaces between "{{" and the template name.
• test—failed to remove comments around the dated AfD template (though it correctly removed the template itself), likely because the line with the last hidden comment did not end in "-->".

Unless they are extremely easy to fix, I do not think fixing the two cases where the template was unable to remove the template is worth your time. I don't think I've ever seen a merge template call that contained spaces between "{{" and the name of the template. Sometimes editors add content in the line containing the last hidden comment, but this is extremely rare and it's really not worth the time it takes to fix it.

You should go ahead! In the meantime I'll continue to use your cute little script and I'll let you know here if I see something go wrong.

[FaviFake]

@NovemLinguae I've used your script to remove a couple dozen more templates, and it works perfectly every time! I'm not entirely sure how you want Phase 2 to be tested, but all we would need to check is whether the list of pages generated by the script is the same as the mainspace and draftspace pages listed in the WhatLinksHere of the AfD subpage + the Target page if the discussion was closed as "merge" or "redirect".

Unfortunately, I'm realising that, due to the way ticket #135 was organised, I forgot to mention that this item should also be part of this patch (I've now removed it from #135 and moved it at the bottom of this message for simplicity):

If and only if the discussion was closed as merge, place {{Being merged from|Source article|discussion=NominationName|date=Date of the closure}} always at the top of the Target page. Otherwise, don't add anything.

To sum up and clarify, this is the full plan for this patch:

1. We always need to remove all merge templates related to this discussion, regardless of how the discussion was closed. Once a discussion is closed, every template is always outdated and always needs to be removed. We are doing this by building the list of pages and running RemoveMergeTemplates on all of them.
2. This second step only applies if the discussion was closed as "merge": in this case, the script also needs to add {{Being merged from|Source article|discussion=NominationName|date=Date of the closure}} at the top of the Target article given by the nominator.

So: remove all old tags and, if the discussion was closed as "merge", put just 1 tag on the Target page.

For example, if Wikipedia:Articles for deletion/PAGE1 is closed with consensus to merge into TARGET, XFDcloser should do these two things:

1. Remove the merge templates related to this AfD from every article. For example, TARGET may have been tagged by Twinkle when the user created the AfD, so we need to remove the tag by running RemoveMergeTemplates on TARGET.
2. Now that we've removed all the tags, we need to tag just the TARGET article because the discussion was closed as "merge": {{Being merged from|PAGE1|discussion=PAGE1 (2nd nomination)|date={{subst:CURRENTMONTHNAME}} {{subst:CURRENTYEAR}}}}

However, if Wikipedia:Articles for deletion/PAGE2 is NOT closed as "merge" (let's say it's closed as "no consensus"), XFDcloser only needs to do 1 thing:

1. Remove the merge templates related to this AfD from every article. Maybe a random page PAGE3 was tagged during the discussion. Since there is consensus against merging, we just need to run RemoveMergeTemplates on the list of pages, just like before.

I admit I should've realised that this step was missing sooner, sorry about that. To help clarify everything, I've moved everything that this patch needs to do below. For the purposes of this patch, XFDcloser only needs to be modified to do the things below and nothing else.

---

i. Build the list of pages to check.

• The list just contains every page in the (Article) or Draft namespaces in the "What links here" of the AfD discussion subpage.
• In addition, if the discussion was closed as merge or redirect, add the Target page to the list, unless it's already on the list or it doesn't exist.

ii. Once you have that list of pages, run RemoveMergeTemplates on each page on the list. (Look at every instance of {{Merge}}, {{merge from}}, {{Merge to}}, {{Article for deletion/dated}}, {{being merged to}}, {{being merged}}, {{being merged from}}, {{Merge portions from}}, or one of their redirects, and remove every one where the Source page or the NominationName appear {{inside it}}. Leave every other one in place, even if it contains the same Target page as this discussion. When removing {{Article for deletion/dated}} or one of its redirects, also remove the hidden comments around it.)

iii. If and only if the discussion was closed as merge (or "merge selectively" or "soft merge"), place {{Being merged from|Source article|discussion=NominationName|date=Date of the closure}} always at the top of the Target page provided by the closer. If it wasn't closed as "merge", do nothing.

[NovemLinguae]

Unfortunately, I'm realising that, due to the way ticket #135 was organised, I forgot to mention that this item should also be part of this patch (I've now removed it from #135 and moved it at the bottom of this message for simplicity):

Can you please edit #135 so that it includes what you removed in the same bullet as the rest of this patch, then take a screenshot and underline/circle whatever you changed/added, then post the screenshot here?

[FaviFake]

@NovemLinguae Done! Here's the explanation of the screenshot below:

• Text highlighted in blue was removed for simplicity but it's still true. For example, I removed a couple of parenthesis that explained the rationale behind a bullet point. These are technically still correct.

• Text highlighted in red was removed because I wanted to simplify the instructions. They aren't incorrect, they're just edge cases that I don't think are worth coding. For example, there's no reason why the script shouldn't run RemoveMergeTemplates if the page is a redirect (this rarely happens, and it would work regardless), so I removed that check. Similarly, I removed the part about always adding the Source page to the list because it's very rare to see an incorrect merge template on the source page. You can still follow these instructions if you want, but I don't think they're worth the time it would take you to code them, so I removed them.

• Moved the last item from the section "AFD - Changes to "Merge" outcomes" to the last section with the other steps. It remained essentially the same, I just moved it.

You can follow both instructions, but if you follow the ones at the bottom of my previous message in this conversation, they are simpler.

[NovemLinguae]

Sigh. That's a bunch of changes. This patch will be done around 2031 I think. lol

[FaviFake]

No, it's basically the same! I only made it simpler and included the last step that I forgot to mention!

That's the only real change: instead of (1) building the list and (2) running RemoveMergeTemplates on it, we (1) build the list, (2) run RemoveMergeTemplates on it, and (3) add {{Being merged from|...}} to the target. You've already done step (2), so only (1) and (3) are missing! Will it really be a big change in terms of coding?

[NovemLinguae]

I see a bunch of algorithm changes to roman numeral 1, which I had already pseudocoded.

And I see the addition of a roman numeral 3, which will change the wikitext algorithm that we already tested. So will need to do some more testing after I change the code.

Placing stuff below the hatnotes means I'll need to add an MOSOrderPositionFinder class, which luckily I have from working on other repos, but adds complexity.

If I recall correctly, this is like the 3rd or 4th revision to the spec after people had started writing patches for it. I hope this is the last change to the spec.

[FaviFake]

@NovemLinguae No no that's perfect, you can continue from the pseudocode you've already written! Don't start over! As I said, both specs are valid and correct. If you've already started working on the original roman numeral 1 or 2, continue from there and ignore my new spec! If you have not started on them, you can choose to follow the new spec, which is basically same but slightly easier to code. The two original roman numerals are simply more robust versions of the new first two bullet points. If you follow the original roman numerals, the script will work on even more edge cases. The last thing I would want is for you to start over, please continue from the first spec if you've already started working on it!!!

Yes, the new "third" roman numeral is the only real addition. I hadn't realised it was missing until now, but it needs to be part of this patch unfortunately. Sorry!

Yes, that's exactly why I removed it! As I said, all of the text I highlighted in in my screenshot (including the light blue text) is stuff I've removed from the spec, not added. The only thing I've added is roman numeral 3. I specifically removed the part that says "below hatnotes" so that you wouldn't need to worry about that! Of course, it would be nice if the templates were added below hatnotes, but it's absolutely not necessary and I wouldn't want you to waste more time on this patch just to code that, so i removed it.

Yes, it is, sorry! As I said, you can discard this new spec completely except the numeral 3. You can follow the original numerals 1 and 2 exactly as they were written, but unfortunately the 3rd one needs to be added.

I think you misread my screenshot. Everything I highlighted was already present in the previous spec and I removed it for simplicity. The colours are not "blue was added and red was removed", both colours mean "removed". I had explained this above the screenshot itself.

[NovemLinguae]

Ah, I see. Green is stuff to be deleted. So if we make green stuff to add, is this an accurate diff?

[FaviFake]

Yes, exactly!

The only change needed is the addition of numeral 3. You can choose to keep or ignore the text you highlighted in red. If you haven't started to work on an item on the list, it's best to ignore it so you have less work to do. If you've already started working on it, you can keep following the red spec.

[FaviFake]

Would it be helpful if I asked on a technical noticeboard if anyone wants to help write the code for this patch? I imagine you haven't started working on roman numeral 3, so I guess someone could develop that part in parallel, while you finish. Or they could do the entire rest of the patch themselves, and we could jump to testing it!

A few months before voorts proposed merging PAM into AfD, I remember DVRTed was developing a script to automatically add the {{merge}} templates when creating a new PAM proposal. Maybe they'd be willing to help out with this one too! What do you think?

[NovemLinguae]

I'm busy for a couple weeks with IRL stuff. Feel free to have others take a look at and help with this patch. If some discussion occurs onwiki, maybe drop a link in this ticket so I can follow along.