XML Converter S3

================================================================
MEMORY FILE — DIRT 2 Binary XML Converter (Python + Nemo Action)
================================================================

Memory name: DIRT 2 Binary XML to Readable XML Converter (Python Port + Nemo Right-Click)
Memory version and date of writing: v1.0 — 2026-09-19
Name of the writer: Deepseek

Relevant sections:
- A0 = everything needed to reproduce the converter from scratch
- B3 = BinXml binary format spec (the reverse-engineering gold)
- B4 = Linux Mint / Nemo right-click install steps
- C2 = the mistake made first (do not repeat)
- C4 = exact end-to-end recipe
- C5 = hard-won lessons

Information ordering rule (disclosed once, before A0, per skill card instructions):
Every section and subsection in this file is ordered from MOST critical to LEAST critical. A0 contains only critical information; if a fact were missing, the goal would fail or the reader would get it wrong.

----------------------------------------------------------------
Section A
----------------------------------------------------------------

A0 — Important or critical information:

1. THE GOAL ARTIFACT is a single Python 3 file (~300 lines, pure stdlib) that converts Ego Engine binary XML ("BinXml" format, used by DIRT 2, GRID, early Codemasters titles) into readable text XML. The user is on Linux Mint (Cinnamon/Nemo file manager). No .NET/Mono dependency allowed.

2. THE GROUND-TRUTH SOURCE is https://github.com/EgoEngineModding/Ego-Engine-Modding — specifically the C# files under EgoEngineLibrary/Xml/: XmlFile.cs, BinaryXmlElement.cs, BinaryXmlAttribute.cs, BinaryXmlString.cs, XmlBinaryReader.cs. The Python script is a direct port of these. The repo does NOT contain a ready-made CLI; the conversion logic had to be hand-ported.

3. THREE DISTINCT FORMATS exist in XmlFile.cs. They must be told apart by sniffing the first bytes:
   - "BinXml"  -> magic bytes 1A 22 52 72 at offset 0. THIS is DIRT 2. Sectioned format.
   - "BxmlBig" -> byte 0x00 then ASCII "BXML". Recursive node format.
   - "BxmlLittle" -> byte 0x01 then ASCII "BXML". Recursive node format.
   - If file starts with "<" after optional whitespace -> already plain text XML, skip it.

4. BinXml SECTION LAYOUT (little-endian; each section is <magic:int32><length:int32> followed by length bytes of payload):
   - Sec 1 magic 0x7252221A — file length
   - Sec 2 magic 0x72522217 — combined length of sec 3+4
   - Sec 3 magic 0x7252221D — NUL-terminated UTF-8 strings (the string pool)
   - Sec 4 magic 0x7252221E — string location table (parser SKIPS it)
   - Sec 5 magic 0x7252221B — N x 24-byte element records
   - Sec 6 magic 0x7252221C — N x 8-byte attribute records
   Every magic must match or the file is corrupt/wrong-format.

5. ELEMENT RECORD (24 bytes, six int32, in order):
   elementNameId, elementValueId, attributeCount, attributeStartId, childElementCount, childElementStartId
   All IDs are indices into the string pool (except attribute*Id which index the attribute array).

6. ATTRIBUTE RECORD (8 bytes, two int32): nameID, valueID. Both indices into the string pool.

7. RECONSTRUCTION RULE (from BinaryXmlElement.CreateElement):
   - Create element named strings[elementNameId].
   - For i in [attributeStartId, attributeStartId+attributeCount): set attr strings[attr.nameID] = strings[attr.valueID].
   - For i in [childElementStartId, childElementStartId+childElementCount): recursively build element at that index.
   - If elementValueId > 0 AND childElementCount == 0: set element.text = strings[elementValueId].
   - Root element is always index 0.

8. OUTPUT STYLE must match .NET's XmlWriter byte-for-byte or nearly so:
   - Declaration: <?xml version="1.0" encoding="utf-8" standalone="yes"?> (lowercase utf-8!)
   - Comment right after: <!--BinXml-->
   - Self-closing tags have a space: <tag ... /> (not <tag .../>)
   - 2-space indent.

9. THE PYTHON SCRIPT MUST be a single self-contained file. No pip installs. Uses only os, re, struct, sys, xml.dom.minidom, xml.etree.ElementTree.

10. RIGHT-CLICK INTEGRATION on Mint/Cinnamon uses a .nemo_action file in ~/.local/share/nemo/actions/, NOT a .desktop file. The Exec= line must use an ABSOLUTE path to the script (e.g. /home/USER/.local/bin/binxml2xml.py) — Nemo does NOT expand ~.

A1 — Short summary of the current chat related to the formal goal:

The user wanted to convert DIRT 2 binary XML files to readable text XML on Linux Mint without installing .NET or the Windows-only Ego File Converter tool. They pointed me at the Ego-Engine-Modding GitHub repo. I initially produced a fictional/guessed parser that was wrong, then the user supplied the actual C# source (using System.Xml;.txt containing BinaryXmlAttribute, BinaryXmlElement, BinaryXmlString, XmlBinaryReader, XmlBinaryWriter, XmlFile, and the XmlType enum). From that, I wrote a correct Python port. We diffed its output against the C# tool's output on a real effects.xml file — they were functionally identical (only cosmetic differences: encoding case, self-closing tag spacing). We fixed those cosmetics, added folder-scan and multi-file support, and set it up as a Nemo right-click action. The final output is byte-equivalent to the original tool.

A2 — Goal of the chat:

Primary goal: produce a working, self-contained Python 3 script that converts Ego Engine BinXml (DIRT 2) files to text XML identical to the C# Ego File Converter's output, runnable on Linux Mint without .NET.

Sub-goals:
- Correctly parse all three sub-formats (BinXml, BxmlBig, BxmlLittle).
- Match the C# tool's output formatting exactly.
- Support batch conversion of every .xml in a folder.
- Name outputs filename(readable).xml to avoid clobbering inputs.
- Integrate as a file-manager right-click action.
- Refuse to overwrite existing (readable).xml files.

A3 — Relevant web sources:

- https://github.com/EgoEngineModding/Ego-Engine-Modding  (main repo; source of truth)
- Path in repo: EgoEngineLibrary/Xml/ (the C# files ported here)
- Path in repo: EgoEngineLibrary/Formats/ (other formats: lng, tpk, pkg — not yet ported)
- Nemo action reference (Linux Mint Cinnamon): ~/.local/share/nemo/actions/*.nemo_action

----------------------------------------------------------------
Section B
----------------------------------------------------------------

B1 — Summary of the entire chat:

The user opened by asking for a Python script to convert DIRT 2 XML to readable XML, referencing the Ego-Engine-Modding repo. I gave a first attempt that was structurally plausible but factually invented — I guessed the binary layout instead of reading the actual source. It didn't work. After the user said "we don't need UIs anyway, right-click actions are better," we pivoted. The user then supplied the actual C# source files from EgoEngineLibrary/Xml/ as a .txt attachment. From those, I wrote a correct port that parses the sectioned BinXml format (magic-tagged sections, string pool, element table, attribute table), plus the BXML recursive format for later Ego titles. We tested on a real effects.xml from DIRT 2. The output was functionally identical to the reference C# tool's output; only two cosmetic diffs existed (encoding case and self-closing-tag spacing), which we fixed. We then added folder-scan mode, multi-file argument support, auto-naming to (readable).xml, and safe-skip logic for already-converted files. Finally, we built a .nemo_action file so it appears in the right-click menu on Linux Mint, replacing the need for Mono/.NET entirely. The user confirmed the final result was accurate and identical to the original tool.

B2 — Your thoughts about the chat or current project:

I made a real mistake at the start: I wrote ~150 lines of code that looked authoritative but was fabricated, because I didn't have the source and guessed. That's the worst kind of failure — confident and wrong. The user caught it by just running it. Lesson: when a file format is involved, refuse to guess. Ask for the parser source, or say plainly "I can't do this accurately without the spec." Once the C# source was on the table, the port was mechanical and correct in one pass. The right-click-action approach the user proposed is genuinely better than a GUI — right call by them. This is the pattern I want to default to: small, dependency-free CLI tools + OS-native integration, not another Electron app.

B3 (custom) — BinXml binary format quick-reference:

Header: 1A 22 52 72 | int32 fileLength
Sec2:   17 22 52 72 | int32 (sec3+sec4 length)
Sec3:   1D 22 52 72 | int32 strLen | strLen bytes of NUL-terminated UTF-8 strings
Sec4:   1E 22 52 72 | int32 locLen | locLen bytes (SKIP: string location table)
Sec5:   1B 22 52 72 | int32 elemLen | (elemLen/24) records x 24 bytes
Sec6:   1C 22 52 72 | int32 attrLen | (attrLen/8) records x 8 bytes

Element (24B, 6 x int32): nameId, valueId, attrCount, attrStart, childCount, childStart.
Attribute (8B, 2 x int32): nameId, valueId.
All IDs index Sec3's string pool (in order, 0-based). Root element index = 0.

B4 (custom) — Linux Mint / Nemo right-click install:

1. mkdir -p ~/.local/bin && cp binxml2xml.py ~/.local/bin/ && chmod +x ~/.local/bin/binxml2xml.py
2. mkdir -p ~/.local/share/nemo/actions
3. Create ~/.local/share/nemo/actions/binxml2xml.nemo_action:

[Nemo Action]
Name=Convert to Readable XML
Comment=Convert Ego Engine binary XML to text XML
Exec=python3 /home/USER/.local/bin/binxml2xml.py %F
Icon-Name=text-x-generic
Selection=NotNone
Extensions=xml;
Quote=double

4. Replace USER with actual username (Nemo does NOT expand ~).
5. nemo -q to restart Nemo. Right-click any .xml -> "Convert to Readable XML".

B5 (custom) — Key deltas from the C# original:

- XmlFile.cs also handles BxmlBig/BxmlLittle via XmlBinaryReader.ReadBxmlElement — ported as convert_bxml.
- .NET's XmlWriter emits <?xml ... encoding="utf-8" ...?> (lowercase) and <tag ... /> (space before slash). Python's minidom emits "UTF-8" and <tag .../>. Fixed with a string replace + regex: re.sub(rb'(?<![ />])/>', b' />', pretty).
- Only writer direction (bin -> xml) was ported. The reverse (xml -> bin) exists in XmlFile.Write / BuildBinXml but was NOT ported. If the user needs to pack edited XML back into binary, that's the next job.
- Python port skips Sec4 (string location table) entirely, same as the C# reader does.

----------------------------------------------------------------
Section C
----------------------------------------------------------------

C1 — Steps taken to reach the goal:

1. User asked for the converter, referencing the Ego-Engine-Modding repo.
2. I wrote a first attempt from memory — WRONG, invented the format.
3. User ran it, got "Conversion not implemented" / format mismatch.
4. I admitted the guess was bad and asked for the actual C# source.
5. User supplied using System.Xml;.txt with the real parser.
6. I ported the C# to Python: parse_binxml, _build_element, convert_binxml, convert_bxml, _pretty.
7. We tested on effects.xml; output was functionally identical, 2 cosmetic diffs.
8. Fixed cosmetics: lowercase utf-8, space before />.
9. Added batch/folder scan, multi-file args, (readable).xml naming, safe-skip.
10. Created .nemo_action for right-click.
11. User confirmed accuracy.

C2 — Mistakes and corrections:

MISTAKE 1 (critical): I invented a fake binary format on the first attempt — wrong magics, wrong structure, wrong everything. Fix: only write a file-format parser after seeing the actual reference source or spec. If asked to do it blind, say "I can't do this accurately without the source" instead of guessing. This nearly wasted the user's time entirely.

MISTAKE 2 (minor): I gave the new main() as a hand-splice patch, which led to a "does not seem to work" report. Fix: after any nontrivial patch, provide the whole file in one block so the user can overwrite cleanly, not merge by hand.

MISTAKE 3 (cosmetic): First working version emitted encoding="UTF-8" and <tag/> instead of .NET's encoding="utf-8" and <tag />. Fix: string replace + regex for self-closing tags; hardcode the declaration line.

USER-SIDE: no real mistakes. The user correctly diagnosed that a .nemo_action beats a .desktop file on Cinnamon, and correctly pushed for the right-click integration over a GUI.

C3 — Proven, unproven and disproven facts:

PROVEN (verified by testing on real DIRT 2 effects.xml):
- The BinXml magic bytes are 1A 22 52 72 at offset 0.
- The six section magics are as listed in A0 section 4.
- Element records are exactly 24 bytes; attribute records exactly 8 bytes.
- Root element is index 0.
- The Python output matches the C# tool's output for effects.xml.
- Nemo .nemo_action with Exec=python3 /abs/path %F works on Linux Mint Cinnamon.

UNPROVEN (plausible but untested in this chat):
- That every DIRT 2 XML file uses the BinXml format (some may use BXML).
- That the port handles malformed/truncated files gracefully (untested).
- That the BxmlBig / BxmlLittle code paths work — no test file was provided.
- That files from GRID / DIRT 3 / F1 share the same format version.

DISPROVEN:
- My initial invented format (bad magics, "chunked" layout, wrong attribute encoding) — do not use.
- The idea that a .desktop file is the right integration on Cinnamon — it's .nemo_action.

C4 — Step by step instructions to achieve the finished goal:

1. Get the C# source. From https://github.com/EgoEngineModding/Ego-Engine-Modding, read every file under EgoEngineLibrary/Xml/. These are the ground truth. Do not start coding until you have read XmlFile.cs, BinaryXmlElement.cs, BinaryXmlAttribute.cs, BinaryXmlString.cs, XmlBinaryReader.cs.

2. Write binxml2xml.py in pure stdlib Python 3. Structure it as:
   - Constants: BINXML_MAGIC = bytes([0x1A,0x22,0x52,0x72]), BXML_MAGIC = b"BXML", six MAGIC_* ints.
   - parse_binxml(data) -> returns (strings, elements, attributes). Read six sections in order, asserting each magic. Skip Sec4 payload.
   - _build_element(idx, strings, elements, attributes) -> recursive ET.Element builder, mirroring CreateElement.
   - convert_binxml(data) -> parse_binxml + _build_element(0, ...).
   - convert_bxml(data, little_endian) -> recursive node reader per ReadBxmlElement.
   - _pretty(root, comment) -> ET.tostring, minidom pretty-print, strip existing decl, re.sub(rb'(?<![ />])/>', b' />', pretty), prepend hardcoded decl + <!--BinXml--> comment.
   - convert_data_to_bytes(data) -> sniff magic, dispatch, return None if already text XML, raise on unknown.
   - readable_name(name) -> base + "(readable)" + ext.
   - convert_one_file(in, out) -> read, convert, write, print OK/SKIP.
   - scan_folder(folder) -> iterate *.xml, skip files with (readable) in name, skip if output exists.
   - main() -> no args = scan script's own folder; single "-" arg = stdout; otherwise treat every argv as an input file, auto-name outputs.

3. Save as ~/.local/bin/binxml2xml.py, chmod +x.

4. Test: python3 ~/.local/bin/binxml2xml.py /path/to/effects.xml -> should produce effects(readable).xml next to it.

5. Verify: diff effects(readable).xml reference_from_csharp_tool.xml -> should be identical or only differ by trailing newline.

6. Install the Nemo action per B4 above.

7. Test right-click: select a .xml, right-click -> "Convert to Readable XML".

C5 — Advice for your future self:

- NEVER guess a binary format. Ask for the parser, the spec, or a hex dump, and refuse to write code until you have one. The single biggest time sink in this chat was a confident, fabricated parser.
- When porting C#, mirror the exact structure: struct fields in order, recursion shape, control flow. Do not "improve" the design; matching behavior is the goal, especially for formats where byte-order and index semantics matter.
- For output-formatting parity with .NET's XmlWriter specifically: hardcode the declaration line, do not trust Python's default casing; and remember <tag /> has a space in .NET, <tag/> does not in Python's minidom.
- Right-click actions on Linux: Cinnamon uses .nemo_action, MATE uses .desktop in file-manager/actions, GNOME uses Nautilus scripts, KDE uses servicemenus. Always use absolute paths in the exec line; ~ is not expanded.
- The "safe-skip" pattern — never overwrite an output file that already exists, and never reprocess files matching your own output suffix — turns a one-shot script into something a user can safely run on a whole folder repeatedly. Adopt this pattern for any batch converter.
- If the user offers to provide the source: TAKE IT IMMEDIATELY. Do not try to be clever and reverse-engineer from behavior. This chat would have finished in one round instead of five if I had asked at turn one.
- The reverse direction (xml -> binary) is not ported. If the user ever wants to mod and repack, port XmlFile.Write / BuildBinXml from the same C# file. Same pattern applies.

================================================================
End of memory file.
================================================================