OCTASOFT

Case study

Cloud-Based Content Processing Framework

A science publishing company needed an auto-scaling batch framework to enrich millions of journals and patents for discoverability.

Industry
Publishing & Research
Duration
27 months
Technologies
Asynchronous content processing framework with an auto-scaling worker poolJournals and patents arrive and are placed on an asynchronous work queue, whose depth drives scaling. A pool of Java and Spring Boot workers draws from the queue and grows or shrinks with the depth of work, producing enriched output for discoverability. A retry path returns failed items from the worker pool to the queue, so individual failures are retried instead of dropped or allowed to stall the run. GitLab CI/CD, running automated JUnit and Cucumber tests behind quality gates, deploys the workers, and Terraform defines the queue, workers and scaling as code.GitLab CI/CDAutomated tests and quality gatesJUnit and CucumberDocuments inJournals and patentsWork queueDepth drives scalingWorker poolWorkerJava and Spring BootScales with queue depthEnriched outputFor discoverabilityFailuresFailed items are retried, not droppedTerraformInfrastructure as code for the queue, workers and scaling
A simplified view of the processing framework. The queue and the retry path are drawn generically because the engagement does not record which services provided them. Illustrative, not as-built documentation.

The challenge

Volume was the obvious constraint; correctness was the harder one. Enrichment that is subtly wrong across millions of documents is worse than enrichment that fails loudly, because nobody notices until researchers cannot find what they are looking for.

The approach

We architected an asynchronous, auto-scaling processing framework on AWS, scaling with queue depth so that a large backlog drained quickly while a quiet period cost almost nothing.

Fault tolerance was designed in from the start. At this document volume individual failures are a certainty, not an exception, so the framework needed to isolate and retry them without stalling the run or silently dropping work.

Testing carried unusual weight here. Comprehensive automated testing with JUnit and Cucumber ran through GitLab CI/CD with quality gates. The alternative was discovering an enrichment defect after several million documents had been processed, which is expensive to unpick.

Infrastructure was managed as code in Terraform throughout.

The outcome

  • Millions of documents processed, scaling automatically with demand
  • Comprehensive test coverage supporting confident, frequent releases
  • Document enrichment processing time reduced by around 75%
  • Materially better discoverability of scientific content for the research community

This was as much a software engineering engagement as an infrastructure one. Java and Spring Boot sat alongside Terraform and CI/CD, and that combination is what made the correctness guarantees possible.

Next step

Facing a problem like this one?

We work on platform engagements where the constraints are real and the outcome is measurable. Describe yours and we will tell you whether we are the right people for it.

Book a consultation