While I have worked less on legacy application prior to Corona times, my understanding of technical debt was around debt and cumulative interest. Yes, we had experiences where to build a quick feature, we have opened debt account, bu we closed debt account, quicker and faster similar to how I handle my credit card.
Yes, This interest can become significant, especially for legacy products with high techdebt. In addition, technical debt accumulates over time and leads to additional costs to maintain and update the product. Found stakeholders have heard technical term debt well and needs more details explanation of the impacts that technical debt had on the organisation in terms of the costs that organisations may incur:
Working with legacy application is difficult for senior person with authority, ready to be hands-on and also interested in technology and solving production issues.
Please find my learning summarised below. at the end,, I wrote down my discovery journey to identify root cause in .NET legacy app with technical debt and product debt .
- Maintenance costs: Cost of maintaining legacy code can range from 50% to 90% of the total cost of ownership of the software. This cost increases significantly as the size and complexity of the codebase grow.
- Legacy products with a high # techdebt require ongoing maintenance, time-consuming and expensive. This includes fixing bugs, addressing security vulnerabilities, and updating the software to remain compatible with new technologies.
- Opportunity costs: Legacy products ties up resources that would be available for better used elsewhere, in absence of technical debt. This lead to miss out on opportunities to innovate and improve their products, that time and money is currently spend to maintain legacy products.
- Security risks: Legacy products may have security vulnerabilities that can put the organization at risk of data breaches, cyber-attacks, and other security threats. Addressing these risks can be costly and time-consuming.
- Reduced productivity: The complexity of legacy products can slow down productivity, as developers may struggle to work with outdated technology and cumbersome code.
- Difficulty finding skilled developers: Lot of legacy products use outdated technology and require specific skills that are outdated in the current job market and it may be difficult to find developers in the current job market. This can make it challenging to find and hire developers who can effectively work on the product.
In addition to technical debt, legacy apps have product debts, documentation debt, performance debts and so on. In summary, the costs of handling typical legacy products with a vast #techdebt can be significant and may impact an organisation’s ability to innovate and stay competitive. It is essential for organisations to consider these costs when making decisions about how to manage their legacy products.
Now comes one root cause identification as part of my multiple journeys to identify technical debts, (on-boarded as leader for 20+ member platform team) The earlier leaders moved out in 2 months of my joining this division, which was from a startup .
Stage 1 – Part A : There are legacy apps that supports platform for live data. Can you take over legacy apps with two fresh engineers who joined your team along with you. They are in training and may be they can help you in this space. The support engineer in your team supports 10 of these legacy application and needs helping hand.
- For some data content, application was too quick
- For some date content, application took quite long time, similar to what was faced in platform
- The challenge was in .NET API and no developer knows .NET expect the support engineer and me.
Stage 1- Part B: Product support team has complained that platform has performance issues for this operation, takes long time. Did some discovery to learn
- Legacy app was .NET Legacy web-service( some claimed micro-service).
- Plaform team is full of non .NET developers. The support engineers who supports legacy apps supports 10 different legacy application including a multiple .NET legacy apps
- The support engineers shared “I am around for 3 years. No one gave me formal KT for the code design and database design. The app was supposed to be decomissioned long back and it has not yet been done”. and assured that “I am aware of the landscape and operational actions required by content team to support legacy application. I know application is slow for 2 years and do not have time to go deep”
Stage 2: Coming to highlighted legacy .NET app, I discussed with manager (non-techie, as all my tech boss have left) who said “ others are gone and you are the leader of the team and want your help to resolve this. We are tight on budget and hence offered two fresh engineers to support you to streamline”. Root cause was projected as platform
Stage 3: The fresh engineers have learnt java and angular. Took help of principal engineer to get API signature . We developed build angular application that involves.NET API and want to evaluate how much time was being taken by API. Observation was clear, the root cause moved to .NET legacy app
- For some data content, .NET legacy app was too quick
- For some date content, .NET legacy app took quite long time, similar to what was faced in platform
- The challenge was in .NET API and no developer knows .NET expect the support engineer and me.
Stage 4 : Discovery working with domain experts, operations team and support engineer
- .NET legacy app used SQL server for database.
- .NET legacy app was used to provide data to consumers and there was .NET utility and excel (handled by product operations team) that saved data to database.
- Larger fun is that my laptop was high-end mac and I have to start with Microsoft visual studio and Mac SQL server tools to get access to SQL database and view what was present.
While I ran around to get access to database operated by my teams’ support engineer, there was more to discover, when to started to list down all the tables in the database and create DB schema.
- There is no separate test environment. Looks like there was one and when database was moved to cloud, they dropped test database to save costs.
- Test tables were created in production database and leverage the same for testing other application, which pushed the data
- 90% of the database size mapped with the storage of test data. No one in team understood DB schema and DB schema showed that there is hierarchy relationship between tables.
- NET code was using C# and was quite old as 2016 or earlier and the servers to host .NET API was windows server 2012 ( this what I see in 2021). Root cause seem to moved to SQL database
Stage 5: Tried to influence business to provide me budget to fix things. The business mangament shared that there was no budget and the domain experts were demanding that this was causing challenge. The more I was able to collaborate with domain experts, they started sharing the actual challenge.
During normal times, when experts pushed data, there was no issue. During delivery time( closure cycles for publishing content and data), some pushed data and had no issue. Some pushed fresh data on the last day, and faced challenge. I doubt whether when multiple push fresh data at same time, performance issue crops up. Did not have evidence still.
Stage 6 I requested experts to include me in upcoming publish cycle at the quarter end. I gave them some guidelines and they were happy to take my guidance.
At the same time, two domain experts will not push data. They would take turns to perform publish data one by one, by posting in slack channel when one starts and when one finished. On seeing start, other expert waited, until the previous finish happened and then performed their publish.
Frustrated with fact that my hypothesis solved issues for domain experts, while I have no clue what was happening, even after being humble and going hands-on to perform L1 support. I tried to diagnose using SQL profiler and found that SQL queries are not taking time. The performance issue was not in the database. While the root cause seemed to be .NET code, I have no clue still where to see
Step 7: With a laptop and other responsibilities, it took time for me to setup .NET code in my laptop and started to walk- through. It was really old code and code looked academic and there were no comments in the code also. Constrained that I cannot debug locally as there was no SQL server on my laptop and neither had .NET environment
This was really a big ball of mud. With other responsibilities as senior manager, I walked through code, thanking Microsoft visual studio Code
- Found at one code before calling database function there is instance of .NET monitor that is used to synchronises access to objects.
- Stateless API has synchronisation object and then it was stateful operation.
- Yes root cause is .NET code. Platform engineers shared that he remembers that legacy windows app got transformed to .NET web service, performance issue in integration exists for long time.
Srini - who am I? 