TEL AVIV — Despite its storybook title, Castle of the Lake is no fairy tale.
It's the code name for the Israeli military's strategic battle-management system, a project developed over 13 years that serves up highly classified data fused from myriad air-, sea-, ground- and space-based intelligence sources to the nation's senior command echelon.
From two underground command centers, one in the so-called pit beneath Israel Defense Forces (IDF) headquarters herein Tel Aviv and another at a base of the General Staff’s C4I Directorate just out of town, Castle of the Lake helps top commanders approve targets for attack, and then assesses how well those attacked targets serve to further overall mission objectives.
And while it's no fairy tale, there is indeed a story of failure, rescue and ultimate redemption behind like its fairy tale-evoking name, Castle of the Lake (Tirat Ha’agam, in Hebrew). is a story, one of failure, rescue and ultimate redemption.
In this story, an Israeli firm called Ness TSG plays the role of hero, the white knight called in to save rescue the project from in-house development failures. The Lotem information technology division of the IDF’s C4I Directorate plays the role of the distressed.
By May, after a decade in industry hands, the story will end on a note of chivalry and redemption when the project reverts totally to IDF hands:
- Redemption on the part of the rehabilitated Lotem IT division and its new Matzpen C2 department, which is now fully empowered to assume control of Castle of the Lake and other strategic projects.
- Chivalry on the part of Ness TSG, which, despite initial protests, is walking away from an estimated $12 million, annually renewable contract free of lawsuits or breach-of-contract claims, according to sources in Israel.
Senior commanders, however, had to be convinced the system should be returned to military control. And Ness TSG officers say this fueled self-doubt by lobbying to keep the program in industry hands.
"It wasn't easy. They didn't like it. There was a lot of lobbying … trying to convince our commanders that it was dangerous; that it wouldn't succeed; that we'd fail again," IDF reserve Col. Avner Ziv, former head of the Matzpen C2 department, told Defense News. "But to their credit, once they understood we were serious, we were ready and that the decision would not be reversed, they accepted it in a very responsible manner and started to work with us on win-win ways to ensure program success."
Moshe Ben-Sha'anan, Castle of the Lake project manager for Ness TSG, said the firm "respected the will of the customer." In a short March 15 telephone exchange, Ben-Sha'anan declined to delve into details of the firm's 10-year effort in running the strategic program, or on the ongoing hand-over process, which started at the end of 2014.
"It's all very sensitive, and I prefer not to discuss it," the executive said. "Suffice it to say we're supporting the process of transferring it back into IDF hands."
In January, state-owned Israel Aerospace Industries (IAI) and Formula Systems, an Israeli-based part of the Polish ASSECO Group, announced its agreement for a 50-50 cash purchase of Ness TSG. The deal, assessed at $50 million, is in the final stages of approval by Israeli regulators and interested stakeholders, said Zvi Ma'ayan, IAI's deputy managing director for business, who is dealing with the pending acquisition.
Neither Ma'ayan nor executives from Formula wanted to comment on TSG's transition out of the Castle of the Lake project.
C4I Directorate commander Maj. Gen. Uzi Moskovitz
Photo Credit: Israel Defense Forces
Searing Failure
It all started in 2004, about a year into in-house development of debut versions of Castle of the Lake. The system prematurely went live for the first time in a General Staff exercise aimed at connecting key users of the IDF high command.
"We had to halt the exercise in the middle. It was declared an operational failure," said Maj. Roi Weizfeld, Castle of the Lake desk director for the Matzpen C2 department, a unit of the Lotem IT division.
At the time, Weizfeld, a graduate of the IDF programming academy, was a newly minted officer working as team leader on a parallel program called Shoval, designed to support high command logistics requirements. In a recent interview, Weizfeld recalled "the debilitating sense of failure" among IDF C4I technologists.
"The system was very slow; performance was lacking ... and this led commanders in our IT division to conclude we were incapable of developing and managing such big strategic projects," he said.
"It was a fundamental setback," Weizfeld said. "It triggered a directive that all major systems had to be outsourced."
A contract was awarded to Ness TSG, whose team of dozens of programmers, engineers and analysts developed, produced, installed and operated the system on behalf of IDF users.
"They did the entire project … from development of the architecture, integration, testing, operation, management. Everything. We, in the IDF, functioned merely as an administrative, contracting office," Weizfeld said.
When asked what went wrong with the in-house program, the officer ticked off a list of offenses: The system was pushed to go live before it was ready. The development process was not properly organized and structured. Changes were not checked and certified as they should have been. Architecture was based on a language that, while advanced in its day, was ill-matched for strategic command-and-control systems and the large number of users they supported. People stayed for a relatively short period and then moved on to other projects. There was no institutional, professional memory.
"At the end of the day, when you get to the point of failure, you realize that we were ill-equipped for this," Weizfeld said.
Lt. Gen. Gadi Eisenkot, right, Israel Defense Forces chief of staff, sits in the command pit supported by the strategic battle management system Castle of the Lake.
Photo Credit: Israel Defense Forces
Glass Ceiling, Rock Bottom
Once the project was handed over to industry, all was well with Castle of the Lake for nearly a decade. Multiple iterations of the system served users through three wars and numerous operations near and far beyond Israeli borders.
But by 2013, new performance requirements, contracting bottlenecks and low morale within the IDF's technology development community prompted users to rethink the added value provided by industrial outsourcing. By then, the system — estimated at nearly $200 million — was suffering from schedule delays and cost growth, officers in Israel said.
And at the same time, "people in this organization were walking around despondent. There was no spark in their eyes. They felt moribund. Everyone complained that all the interesting work was being done outside," Weizfeld said.
After Castle of the Lake was found lacking in a critical 2013 General Staff exercise with the IDF's Northern and Southern commands, users began mulling options for transitioning the high-priority project back into IDF hands.
"We hit a glass ceiling in terms of the program," Weizfeld said. "And we hit rock bottom in terms of internal morale."
According to the officer, the system had essentially maxed out its growth potential. It couldn't support high-echelon commanders who demanded full situational awareness of evolving operations, whether in Gaza, south Lebanon or much farther beyond Israel's borders. Additional users or new capabilities proved very time-consuming and costly to integrate into the system. Functionality was low. It took "two or three minutes to refresh. Each action of the system took several long seconds," Weizfeld said.
Changes or upgrades to the system often took two or more years, with one seemingly straightforward requirement of adding capacity for 200 new users morphing into a complex and expensive 18-month process, officers said.
"The subject of scalability was extremely problematic," Weizfeld said. "Every three years we had to replace the servers. This led to huge outlays."
Moreover, officers said the system became so cumbersome that it took several weeks to train reservists needed during drills or critical operations.
"The concept of operation was very complex, with lots of buttons, unnecessary data displays and many actions needed to get to the desired result. Everything was overspec; it was crazy, and it demanded a massive and onerous training process," Weizfeld said.
After Castle of the Lake was found lacking in a critical 2013 General Staff exercise with the IDF's Northern and Southern commands, users began mulling options for transitioning the high-priority project back into IDF hands.
Photo Credit: Israel Defense Forces
New Beginnings
Now the IDF and Ness TSG are working hand in hand to return full program responsibility to the Matzpen department.
The first phase of the handover was implemented in 2014, after Israel's 50-day war in Gaza. The second phase took place in January, with integration of an in-house developed upgrade that was used to support a major General Staff drill with IDF Northern Command.
The in-house upgrade took less than one month "from the moment the major general outlined on the blackboard for us what he needed until it was uploaded and operational in support of the January exercise," Weizfeld said.
By the end of April, the handover process should be complete, according to the IDF.
"The plan is by [the Jewish holiday of Passover], we'll take final receipt from the company of all that it developed and merge it with integration of all that we developed," Weizfeld said. "It's friendly, but still sensitive and a bit tense. … But we need to note that the company did amazing work over a decade. It brought us to where we are today."
Like earlier versions, in-house versions of the upgraded system are encrypted with redundant, cyber-proof, user-authentication safeguards developed in house by another unit of the IDF's Lotem IT division. But the new Castle of the Lake now operates off a commercial server and is based on commercially developed, off-the-shelf languages that are easier to install, operate and use.
"We don't have the problem of training anymore. The system is simple, streamlined and fully intuitive; it's the same feeling users have when operating Facebook or Google," Weizfeld said.
According to Castle of the Lake program manager, the system opens in less than a second, as opposed to the 30 to 40 seconds it took in earlier TSG-developed versions. Whereas before integration of new codes had to be done manually, the new system is automatic.
"What took three to four hours now takes 7.48 minutes. Each time a programmer changes something or writes a line of code, the system is built to update itself automatically to the server," Weizfeld said.
From an added-value operational perspective, the system no longer monitors merely the number and types of targets attacked by IDF forces, but it prioritizes and grades them in terms of how well their destruction contributes to overall mission objectives as defined by the high command.
Ziv, the former Matzpen commander, estimates that the IDF will accrue significant savings annually as a result of reclaiming its C4I prime contractor status.
"This project, over the years when we depended on industry, amounted to hundreds of millions of shekels. And if in those years we spent about 40 million to 50 million [shekels] annually, I think we'll be looking at annual costs of between 25 percent to 30 percent less than that," he said.
Yet, Ziv insists that projected savings are a bonus. The real benefit of recapturing in-house development control, he said, is the ability and agility with which the IDF can satisfy its own operational requirements.
According to current and former commanders, the IDF has internalized voluminous lessons learned from earlier failures and has changed its culture and work processes sufficiently to prevent future setbacks.
People are now hand picked and often remain with specific programs for up to 10 years at a time. The Lotem IT division — of which the Matzpen department is a part — is now on the lookout for talent as early as the ninth grade and encourages prospective inductees prior to military service. It also cultivates Lotem division alumni now in the high-tech industry to mentor young soldiers and officers throughout their military careers.
The division recently started what Lotem's commander, Brig. Gen. Daniel Bren, calls a "Young Turk" program for creativity and innovation.
"We'd like to be more entrepreneurial in the way we tackle problems and meet operational requirements," Bren told Defense News.
Castle of the Lake, he said, is just one example of high-value, strategic programs that from now on will be internally developed via the C4I Directorate. His Matzpen C2 department now serves as prime contractor for two new command-and-control systems: one dealing in cyber and the other in the electromagnetic spectrum.
And officers say claim that, as of now, both are making good progressBoth, officers claim, are making good progress. Yet, as far as this story goes, only time will tell.
Email: bopallrome@defensenews.com
Opall-Rome is Israel bureau chief for Defense News. She has been covering U.S.-Israel strategic cooperation, Mideast security and missile defense since May 1988. She lives north of Tel Aviv. Visit her website at www.opall-rome.com.
