Dialogue of AI is throughout us, however in my expertise, sensible steering rooted in particular use circumstances is surprisingly uncommon. After spending months deep within the weeds of a large documentation migration with AI as my assistant, I’ve realized some hard-won classes that I feel others may benefit from.
For those who work in content material engineering, technical documentation, or are merely inquisitive about how AI holds up in a fancy, real-world undertaking, right here’s my tackle what labored and what didn’t.
Challenge Context
I’m a DITA Info Architect on the Info Expertise staff at Splunk. DITA, brief for Darwin Info Typing Structure, is an open, XML-based customary for structuring and managing technical content material.
We just lately wrapped up the migration of three massive documentation websites right into a single assist portal, powered by a DITA-based part content material administration system (CCMS). The timeline was tight, and almost all the assets had been inner. The migrations had been complicated and vital to the enterprise, requiring cautious planning and execution.
I initially deliberate solely to assist the migration of the smaller, unversioned website. When that went nicely, I used to be requested to steer the a lot bigger second migration. (The third website was dealt with by one other staff.) Collectively, these two migrations meant grappling with roughly 30,000 HTML information, two very totally different website architectures, and the problem of customizing an current Python migration script to suit the content material at hand, whereas additionally placing processes in place for writers to assessment and clear up their content material.
I need to be clear that AI didn’t full this undertaking for me. It enabled me to work sooner and extra effectively, although solely whereas I did the planning, architecting, and troubleshooting. Used successfully, AI turned an influence device that dramatically sped up supply, but it surely by no means changed the necessity for experience or oversight.
All through this undertaking, I used the then-current GPT-4 fashions by means of an inner Cisco chat-based deployment. Today, I work extra in editor-based instruments resembling GitHub Copilot. Nonetheless, the teachings I realized ought to apply to the current (mid-2025) state-of-the-art, with a number of caveats that I point out the place related.
How I used AI successfully
Prompting
One lesson I realized early on was to deal with prompts the way in which I strategy technical documentation: clear, constant, and complete. Earlier than consulting the AI, I’d sketch out what wanted to occur, then break it down into granular steps and write a immediate that left as little to the creativeness as attainable.
If I wasn’t positive in regards to the resolution, I’d use the AI as a brainstorming accomplice first, then observe up with a exact immediate for implementation.
Iterative improvement
The migration automation wasn’t a single script however turned a set of Python instruments that crawl navigation timber, fetch HTML, convert to DITA XML, break up matters into smaller items, map content material, and deal with model diffs. Every script began small, then grew as I layered in options.
I shortly realized that asking AI to rewrite a big script unexpectedly was a recipe for bugs and confusion. As a substitute, I added performance in small, well-defined increments. Every function or repair obtained its personal immediate and its personal GitLab commit. This made it simple to roll again when one thing went sideways and to trace precisely what every change achieved.
Debugging
Even with good prompts, AI-generated code not often labored completely on the primary strive – particularly because the scripts grew in dimension. My handiest debugging device was print statements. When the output wasn’t what I anticipated, I’d sprinkle print statements all through the logic to hint what was occurring. Typically I’d ask AI to re-explain the code line by line, which regularly revealed refined logical errors or edge circumstances I hadn’t thought of.
Importantly, this wasn’t nearly fixing bugs, it was additionally about studying. My Python expertise grew immensely by means of this course of, as I compelled myself to actually perceive each line the AI generated. If I didn’t, I’d inevitably pay the worth later when a small tweak broke one thing downstream.
Today, I lean on an AI-powered built-in improvement atmosphere (IDE) to speed up debugging. However the precept is unchanged: don’t skip instrumentation and verification. If the AI can’t debug for you, fall again on print statements and your individual means to hint the issue to its supply. And at all times double test any AI-generated code.
AI as an implementer, not inventor
This undertaking taught me that AI is improbable at taking a well-defined thought and turning it into working code. However should you ask it to design an structure or invent a migration technique from scratch, it’ll most likely allow you to down. My most efficient workflow was to (1) design the method myself, (2) describe it intimately, (3) let the AI deal with the implementation and boilerplate, and (4) assessment, take a look at, and refine the AI output.
Model management
I can’t stress sufficient the significance of model management, even for easy scripts. Each time I added a function or mounted a bug, I made a commit. When a bug appeared days later, I might stroll again by means of my historical past and pinpoint the place issues broke. Positive, that is primary software program engineering, however once you’re working with AI, it’s much more essential. The rate of change will increase, and your individual reminiscence of every modification is inevitably much less exhaustive.
The web impact of those practices was pace with out chaos. We delivered far sooner than we might have in any other case, and the standard of the output considerably decreased post-migration cleanup.
The place AI fell brief
As beneficial as AI was, it had many shortcomings. The cracks began to point out because the scripts grew in dimension and complexity:
- Context limits: When scripts obtained longer, the AI misplaced observe of earlier code sections. It might add new standalone options, however integrating new logic into current, interdependent code? That always failed except I spelled out precisely the place and find out how to make adjustments. I ought to notice that at this time’s newer fashions with bigger context home windows would possibly cut back a number of the points I bumped into with the migration scripts. However I believe that it’s nonetheless vital to be as particular as attainable about what sections must be up to date and with what logic.
- Failure to discover a working implementation: I discovered that typically the AI merely couldn’t remedy the issue as outlined within the immediate. If I requested for a change and it failed three or 4 occasions, that was often a sign to step again and check out one thing totally different – whether or not that meant prompting for an alternate strategy or writing the code myself.
- System understanding: Sure bugs or edge circumstances required a strong understanding of our techniques, like how the CCMS handles ID values, or how competing case sensitivity guidelines throughout techniques might journey issues up. This can be a essential space the place AI couldn’t assist me.
What I’d do in a different way subsequent time
Right here’s my recommendation, if I needed to do it another time:
- Plan core libraries and conventions early: Determine in your stack, naming schemes, and file construction on the outset and embrace them in each immediate. Inconsistencies right here led to time wasted refactoring scripts midstream. That mentioned, working in an editor-based device that’s conscious of your whole pipeline will assist to maintain your libraries constant from the outset.
- Sanitize all the pieces: File names, IDs, casing, and different seemingly minor particulars may cause main downstream issues. Embrace this steering in your prompting boilerplate.
- Account for customized content material: Don’t assume all docs observe the identical patterns and positively don’t assume the AI understands the nuances of your content material. Discover out early the place the outliers are. This upfront work will prevent time in the long term.
- Doc the complicated stuff: For any logic that takes various minutes to know, write down a radical rationalization you possibly can refer again to later. There have been occasions I needed to re-analyze sophisticated components of the scripts weeks later, when an in depth notice would have set me again heading in the right direction.
One non-AI tip: hold copies of your supply and transformed markup in a repository even after importing the transformed content material to your manufacturing tooling. I promise that you simply’ll must refer again to them.
AI as a accomplice, not a alternative
Reflecting on the undertaking, I can emphatically say that AI didn’t substitute my essential considering. As a substitute, it amplified my expertise, serving to me work at a pace and scale that might have been troublesome to realize alone, whereas streamlining the post-migration cleanup. However anytime I leaned too closely on AI with out cautious planning, I wasted time and needed to backtrack.
The true worth got here from pairing my area information and significant considering with AI’s means to iterate shortly and implement. Used thoughtfully, AI helped me ship a undertaking that turned a profession milestone.
For those who’re dealing with your individual daunting migration, or simply need to get extra out of AI in your workflow, I hope these classes prevent some ache, and perhaps even encourage you to tackle a problem you might need thought was too huge to deal with.
Discover extra tales on our Innovation channel and subscribe right here!
