What are the implications of more mobile, competitive government agencies?

Last time, we looked at private sector business models around the mobility concept. Today, we will look at the public sector. Upfront, let me assure you this is not a partisan manifesto.

To a large degree, the same transformation I described for the private sector could occur in the public sector. However, the public sector usually lacks the same profit motivation.

Many daily experiences combine private enterprises with government functions. Every time you get a business license, hire an employee, verify background or employment, check on a transfer payment recipient, pay your taxes, or submit a regulatory report, you’re combining the two. For example, in the US, the e-file tax system connects the filer (or filer’s accountant), the federal and state governments, and the bank-managed ACH payment system.

Moreover, formal public-private partnerships tightly link the two worlds, despite their separate motivations.  There is no reason why their historic focus around utilities, transportation, and health infrastructure could not extend to the IT world.

The government may charge a premium for these IT services. But many citizens would find it valuable and desirable to have them managed by highly-regulated body less motivated by profit. Maybe we need a government’s bill of rights that allows new ways to make money from services, beyond taxes. The current budget could certainly use it.

Until recently, for example, the US Postal Service (a semi-governmental agency) actually turned a profit. Legally, the Postal Service must remain revenue-neutral. But last year, they tallied a $10 billion loss. Sen. Claire McCaskill aims to fight this with a “write more letters campaign,” and the USPS itself is running ads claiming “a refrigerator has never been hacked.” Imagine if the USPS was empowered to be more competitive and try a variety of things. For example, they could provide special purpose vehicles (SPV) for managing email or payment service. Some state-owned European postal services successfully offer this.

For folks worried about Big Brother government abusing your data, privately-owned corporations face their own data privacy problems. More than 100 million T.J. Maxx customers were affected by an 18-month identity theft operation ending in 2007. In recent weeks, Zappos admitted that data for its 24 million customers has been compromised.

Legal ramifications for reputation and, ultimately, profits may encourage private companies to guard against data loss. But the Patriot Act and other post-9/11 regulations abroad have already extended the government’s reach into private sector data.

Regardless of sector and use case, all of these integrated, cloud-based business models and processes rely on the same thing: shared master data! This includes the full spectrum of master data: all domains, types (transactional and analytical), high and low latency.

However, we must consider the dark side of any public-private experiments. If you’re scared by Facebook tracking behavior after log-off and indefinitely keeping your history, think what a government agency with a private-sector offering may mean. To compete, such offerings will likely maximize their leverage in the legislative process and may lack the necessary oversight.

Consider a new class of “robber barons” like Carnegie, Rockefeller and Astor. Rather than controlling coal mines and steel mills, they control key steps in the information flow between government and the business world. Their goal is to drive the most profitable steps in the delivery of services and products.

They control what data you see and use, as well as the processes and standards to do so. They will know everything about you, and like Max Headroom, they’ll use this to sell you the next gadget. They’ll use regulation to maintain their advantage.

On the bright side, these IT barons would likely be forced into more transparency, rather than buying political influence, given the formalization of the structures and that state of social media today.

It will be a cool but scary world. But it’s less scary if you know what you’re doing. You could even wipe yourself off the grid.

How can we use Columbo's methods for non-obvious relationship awareness? Jarrett explains

The late Peter Falk played Lieutenant Columbo, the iconic television detective from the 1970s. The cigar-smoking, trench-coat wearing Columbo was shabby and unassuming and his murder suspects – usually the rich and powerful living in Los Angeles– would initially judge the book by its cover. But Columbo, bumbling as he was, was a brilliant strategist. His line of polite questioning eventually unraveled the complex schemes his suspects had crafted. (As he was about to leave, he would say, “Oh… just one more thing…” which became his famous catchphrase.)

The point was not discovering how the murder was conducted: viewers usually witnessed the prologue and aftermath of the crime. The point was watching Columbo put the pieces of the crime together through logical reasoning, miscellaneous clues and contradictory statements from his suspects. No matter how confident the perpetrator, no crime was perfect, and we were able to watch a true master at work.

Data analysts facing data quality errors operate much like Lieutenant Columbo. They start with a hypothesis about what went wrong with the data (i.e., what "killed" it) and then take the steps to weed out the suspect.

As it relates to data systems, this means going back in time to pinpoint the scene and source of the crime. As it relates to business owners, this entails diplomatic questioning to determine which business rules were involved in the data manipulation and the exceptions that might have been involved.

Once the suspect has been targeted, like Columbo, the data analysts must take the difficult step to prove the source of the errors to others, to make even the most skeptical stakeholder say, "Yes, I see it now". Due to politics and territorial pride, this step often has its own perils, but determined data analysts must be willing to pursue this path, despite the challenges presented by those around them.

This type of detective work is a technique not just limited to human beings. One use of suspect identification can be found through Akinator, an entertaining version of the classic car trip game called 20 Questions. I'm not exactly clear how the process works on this one, so curious readers might want to try it out and shed some ideas in the comments below.

Another impressive way to weed out criminals employs a technique called Non-Obvious Relationship Awareness (NORA). NORA enables systems to find real-time relationships amid the stream of name inputs, credit card transactions, terrorist watch lists, monetary transfers, and so on. When a conflict of interest among relationships (e.g., bad guy records commingled with good guy records) is found among otherwise unrelated data, the system sends an alert to the appropriate authorities.

NORA has proven quite effective in recent years. For example, IBM Infosphere Identity Insight has been used to save institutions millions of dollars in lost revenue due to money laundering schemes, credit card fraud, and tax evasion. In these studies, NORA has become a financial Columbo, combining bits of seemingly irrelevant data to systematically solve crime.

NORA is just the next evolution and there is more to come. The world moves faster than ever and computers will continue to support this effort on a grander scale. The work of interrelationships for the data detective undoubtedly will become easier over time and, with each new leap in technology, there will always be "just one more thing."

Editor's Note: This is Jarrett's last post on Mastering Data Management, as he is pursuing other opportunities outside of IBM. You can continue to follow him on Twitter @jgoldfed.

Illustration courtesy of Chris Kotsakis.

Should we use the SSN as an identifier? Or is there a better way?

Entity resolution requires strategy, particularly in weeding out duplicate records stored within the same data system. The very nature of finding these duplicates requires commonality among certain attributes like full names, addresses, birthdates and gender. For the most part, these attributes are inconsistent and require specialized algorithms to help determine a level of confidence.

For example, full names can often differ among two records (Jimmy, Jim and James). Addresses change due to moving. Birthdates are often missing. Even gender can be up for debate. (Is “Chris” equivalent to Christina or Christopher?)

The only attribute that does seem to provide a level of confidence, at least for US populations, is the Social Security Number (SSN). The SSN is often the most critical attribute to determine duplication, so much so that several of the healthcare projects with which I have worked assume the same SSN in two different patient records to be synonymous with duplication, despite any other potential unique identifiers associated with those records.

The Social Security Number came into existence in 1936 and was intended to be used for tax purposes only. In fact, in 1946 until 1972, the legend "FOR SOCIAL SECURITY AND TAX PURPOSES -NOT FOR IDENTIFICATION" appeared on each card. Thus, the tradition of using the SSN to distinguish among US patient records appeared to be one performed out of convenience rather than design.

This strategy worked well in a pre-Internet world, but it is slowly becoming obsolete. Private information has become increasingly easy for hackers to access and SSN-only fraud makes up the majority of cases in identity theft.

SSN as a valid authenticator has been so disconcerting that, in 2002, IBM requested that the more than 100 companies offering health insurance to its workers refrain from printing Social Security numbers on their insurance cards. The handful of health insurance companies that responded to this request replied with a blunt "no."

In several other countries, such as Denmark and Australia, the use of a national healthcare ID has made some traction that is being scrutinized by other countries.

The U.S. military has begun to take some initiative on this front.  In response, to congressional mandate, the DoD has begun issuing all U.S. military personnel a new identification card with a unique reference number. This value, known as the DOD EDI PN, will be printed on the face of the card in place of the SSN, effectively serving the identification purpose that the SSN was never intended to have.

More importantly, because the DOD EDI PN is associated with an individual role and does not have dual-purposes, such as those associated with credit card and banking information, the potential risks associated with exposing a DOD EDI PN to the general public is greatly reduced.

The goal of establishing a unique number for all DOD personnel will take several years and there will be undoubtedly be some bumps along the way. In other private sectors, the use of the Social Security number does not appear to be going anywhere soon. Too many legacy systems use it for finding uniqueness, and it is still the best identifier until a new and inexpensive standard comes along.

Until that day comes, we will continue to provide our private information across open networks, avoiding the bad guys as best we can.  It is ironic that in our social age, our Social Security number is anything but secure.

What do you think? Should SSN be eliminated and if so, how could this be done on a national level?

Jeff's Rule #1 for Successful Identity and Social Network Analysis: Assume no clue is too small

While you may have a different term for it, Identity and Social Network Analysis is a technique for uncovering and analyzing relationships. It takes the form of software designed to look at entities—defined as unique persons or institutions—and analyze their relationships to detect collusive activities.

With Identity and Social Network Analysis, you can see relationships both within and outside of your organization, showing the extended social, business or relational network of an entity.

As we think and talk about the role of Identity and Social Network Analysis specifically in fraud applications, I’ve starting coming up with rules for success. Recently, we described these in a new whitepaper, Preventing fraud with identity and social network analysis: A guide for bank executives.

The short version …

1. Assume no clue is too small. You have to assume fraudsters are actively trying to hide. Thus the algorithms and approach to match analysis and pattern detection must be designed to derive insight from large amounts of sparse, seemingly innocuous data. Many clues may seem insignificant on their own but yield great meaning when viewed together in a larger context. Frankly this is the most important point: everything else depends on doing this well.

2. Operate in real time To prevent fraud, your identity and social network analysis tools must operate in real time or near–real time. You can’t wait for a new load of data to process in a couple days. Undoing the effects of fraud after the fact can be far more expensive than prevention.

3. Move beyond transactions to people and groups.  Stated in a Zen-like way, you bring the transaction into context with the person and their group activity. Only then does the person or group of people have meaning. Likewise, the transaction may be valid but the knowledge of who it really came from and who they are related to gives it meaning.

4. Look for links across channels. Fraud can be sophisticated, an intricate coordinated set of activities across difference touch points with the business. This means data will be distributed, will be in different formats, and can structured or unstructured. Yet the success of your system depends on using this data. The best solutions will recognize this and embrace it.

5. Adapt to changes in fraud tactics. In the world of Information Security, attackers will adopt new tactics every time a “threat vector” is shut down. Likewise, fraud tactics will change. The best identity and social network analysis solutions must have the built-in capability to adapt but not require you to reload and start over.

What has your experience been? Is there another rule that we must include?

Download the white paper: Preventing fraud with identity and social network analysis: A guide for bank executives.

A Master Student Index can help track education success across classrooms and states

In my last post, I explained how a Master Student Index offers a less-invasive, more cost effective approach for sharing student data across institutions. States that look to implement a Master Student Index should consider multiple factors to ensure that they invest in a solution that can meet both their current and future requirements.

Consider these criteria for evaluating Master Student Index solutions:

Accuracy: A Master Student Index must be able to accurately identify students in spite of variations, misspellings, and other errors. Modern algorithms use statistical analysis to determine whether two records refer to the same student. Choose a Master Student Index that has sophisticated algorithms for identifying students. Predictive analysis will only be as good as your ability to accurately link student data.

Flexibility: The first generation of statewide student IDs and longitudinal data systems only had to incorporate data from K-12 systems. The next generation expands these to incorporate preschool and postsecondary institutions. Eventually, these systems may expand to incorporate workforce data and/or corrections data. A Master Student Index needs to be flexible enough to take on new sources of data and to use new data attributes from those new sources to identify students.

Non-invasive: A Master Student Index should not place unnecessary burden on participating institutions. It should offer a number of ways to integrate with their legacy systems. It should recognize that each system will have its own format for storing data.

(Some systems might store a name as a first, middle, and last name. Other systems might store data in a single name field.) A non-invasive Master Student Index will accept data in any format and still accurately identify and cross-reference students.

Secure: The idea of storing even a cross-reference of student identifiers in a single place races legitimate privacy and security concerns. A secure Master Student Index will use role-based access controls to limit the use of data to only authorized users. It will also maintain an audit trail of where data came from. It will record when, how and why data was linked together. And it will log who has accessed what data.

Master Data Management and Master Indexes

Those who are familiar with the concept of Master Data Management will recognize this framework of a Master Student Index. Healthcare providers and regional and statewide health information exchanges use Master Patient Index solutions to allow doctors to share patient information to deliver more effective care. Social service agencies use Master Citizen Index solutions to coordinate benefits and improve customer service across separate agencies.

The good news is that this concept of a Master Index is not new. Departments of Education can leverage lessons learned from others who have solved similar problems. Thus, we’ll get ever closer to being able to analyze data and answer questions that will help us understand the factors that contribute to student success, and to using those answers to improve education for our children.

Catch up on the first two posts in this series: Measuring Education Success with Longitudinal Data Systems and From K to 12 and Beyond: Tracking Student Progress

How should schools track student progress?

In my last post, I discussed how longitudinal data systems are intended to support data-driven decisions to increase student achievement and close achievement gaps. Race to the Top encourages states to expand these systems to include additional programs and data. Race to the Top further encourages states to work together to use longitudinal data systems across states.

Yet states face the challenge of expanding these systems and coordinating with these diverse stakeholders in the face of austere budget constraints.

With these challenges in mind, let’s consider two alternate approaches for expanding longitudinal data systems to share data and analyze student progress across a student’s educational career.

Option 1: Establish a Statewide Student ID

The first possible approach to sharing data across these institutions is to establish a statewide clearinghouse that assigns a unique, statewide student ID for every student in the state. That ID would follow the student from institution to institution, so that progress and results could be tracked across programs. In fact, many states have already implemented this approach as part of establishing K-12 longitudinal data systems.

The challenge with this approach is that as longitudinal data systems expand to incorporate preschool and postsecondary institutions, the burden falls on these other institutions to change or enhance their IT systems to accept, use and report using a new student ID.

Furthermore, as more and more programs create and use a common identifier, there is more opportunity for errors and misuse. Jon Smith’s identifier gets mistakenly used for Jonathan Smith, even though they’re different students. Sarahi Valente gets assigned a different identifier from Sarai Valente Mendez, even though they’re the same student. Also, statewide student IDs won’t help states share data or longitudinal data systems with other states.

Option 2: Create a Master Student Index

Rather than force legacy IT systems to use a new identifier, a less invasive approach is to deploy a master data management solution called a Master Student Index.

A Master Student Index establishes a registry or reference of students and the systems that house their data. A Master Student Index knows that Sarai Valente Mendez is known by the identifier “123456” in the Washington County School District, but known by the identifier “456789” to Washington Technical College.

Neither of these systems need to accept a new identifier or even to know what other systems have data about a student.

The Master Student Index maintains the cross-reference of student identifiers and can even relate these identifiers to existing K-12 statewide student IDs. Any institution that reports student data can provide their existing, local student IDs and let the Master Student Index relate the data to the right student. Similarly, a Master Student Index allows multiple states to share data without requiring a common student identifier.

This Master Student Index approach results in fewer changes to existing IT systems, meaning that this approach is less expensive and reduces barriers to adoption by the institutions that need to cooperate.

In my next post, I’ll discuss capabilities that you should look for to identify a Master Student Index that meets your needs.

Translating can lead to data management problems

As the international community worked to freeze the assets of Muammar Gaddafi they ran into a problem. How do you spell it? Is it Gadhafi, Gaddafi, or Qaddafi? And then they realized there are over 100,000 different ways to spell the name.

The problem is transliteration. Converting Arabic names into a Romanized form varies depending on who did it and in which country. There is no standard spelling. Check out this image from the Wall Street Journal.

How do you search for and freeze the assets of a person whose name you don’t really know how to spell?

It is impractical to search on every variation of the name because almost certainly you will misspell it or miss some variation. You need technology to be able to make this conversion for you.

Ideally you should be able to submit a name – Muammar Gaddafi – and the search system automatically understands the cultural derivation and all the other possible variations. It finds Gheddafi, Qadhafi, Al-Gaddafi, El-Qaddafi, as well as the variations in the first name and multi-part variations.

This isn’t really a data quality problem, as these are all perfectly valid ways to spell the name.  Algorithms that search on the text won’t do either for obvious reasons.

Technology like IBM’s InfoSphere Global Name Recognition is the answer.  InfoSphere Global Name Recognition (GNR) helps the name search problem by applying a unique linguistic-based approach to the name that determines things like the gender, cultural heritage, and proper parsing. Its approach allows you to find the name variations that come as a result of transliteration.

Freezing the assets of Gaddafi is a great example of the multi-cultural name problem. Do you have any interesting examples?

Editor's Note: Check out Jarrett Goldfedder's recent Doggy Data Quality for another perspective on naming conventions.

Can you use educational data to predict graduation rates?

Are children that attend Head Start programs more likely to graduate from college? Are students in accelerated learning programs more likely to succeed in the workforce? Are there early warning signs that predict whether a child is likely to drop out before graduating high school?

Questions like these are surprisingly difficult to answer. The data required to answer these questions are spread across multiple systems owned by different organizations. Individual schools, school districts, and postsecondary institutions all have their own data systems. Other challenges contribute to the difficulty of analyzing student progress across a student’s educational career:

• Students don’t always live in the same place for their entire educational career. They move within a school district, across school districts to other areas of the same state, or even across states. Some students move out of state for a time, only to return years later. This means that data for a student will wind up in different systems across a state or even across the country.
• A student’s name and information could differ across systems. Alex Eastman living at 11705 Oak Lane in one system could be Alexander Eastman living at 11705 Oak Ave in another. People use nicknames, names get misspelled, typographical errors occur, and as mentioned, students’ addresses change.

The Educational Technical Assistance Act of 2002 encouraged states to establish longitudinal data systems to address some of these challenges. Longitudinal data systems are intended to manage, analyze, and use data such as student records, to make data-driven decisions to increase student achievement and close achievement gaps.

Many states have started longitudinal data systems by collecting data from Kindergarten through 12^th grade (K-12). Unfortunately, these longitudinal data systems still lack the breadth and depth of data they need to answer questions like those posed above.

Expanding Longitudinal Data Systems

The Race to the Top program announced in 2009 encourages states to expand their use of longitudinal data systems to incorporate additional programs such as early childhood programs, English language learner programs, at-risk and dropout prevention programs, and postsecondary education.

As states look to expand their longitudinal data systems they face additional challenges.

First, states must collaborate with not just public, but also private educational institutions (preschools and universities). This means that states will have less direct control over how these other institutions cooperate to achieve public goals. Solutions that are invasive or require a lot of effort and expense from these third parties will hamper cooperation.

Second, states are facing austere budget constraints so they must solve the problem as much as possible with existing systems and investments.

In the next post, we’ll look at a couple of options for expanding longitudinal data systems with these challenges in mind.

Insight into your data can detect fraud, protecting your customers and bottom line

It’s been a very busy few months as my role has evolved from days at Initiate Systems to my role as Program Director, Product Marketing for the IBM InfoSphere MDM portfolio. I now have   responsibility for several best-of-breed MDM products, and a product that goes beyond master data management – IBM InfoSphere Identity Insight.

Traditional MDM products focus on uses where an organization is trying to understand people or products or accounts or other data and information that are usually not hidden or masked. That’s not the case with Identity Insight. Rather, it is used to discover the use of false identities and networks of individuals trying to hide their relationships to each other.

Originally developed to identify collusion in Las Vegas casinos, Identity Insight quickly found a strong presence in the public sector marketplace, specifically the intelligence community and law enforcement. Such insight helps locate individuals with nefarious plans and is used in link analysis and possible threat alerts, while also developing master person index and analytics for the law enforcement community.

The same process has been applied successfully to financial and insurance fraud analytics. Using the same capability that finds criminals and terrorists, we can find fraudsters, fraud rings and their associated activities. And, increasingly, within social services and taxation, this technology provides a platform for benefit verification, determining when a person deserves benefits they’re not receiving, or when a person is trying to take advantage of unqualified benefits.

Identity Insight answers three types of questions that we need to answer when dealing with data: Who is who? Who knows who? And who does what? Answering all three questions provides valuable insights that can be used to detect fraud and threats.

Within IBM, these capabilities define our Identity Insight product. Answering all three of these questions delivers valuable insight that companies in a variety of industries are using to detect fraud and threats.

MoneyGram International is doing a phenomenal job at this, using IBM to prevent millions of dollars in fraud to date. MoneyGram International, Inc. is a global payment services company based in the United States that provides global money transfers, money orders and payment processing solutions for individuals and financial institutions.

MoneyGram posted a press release about their efforts to comply with new regulations and help mitigate fraud, citing several statistics about their efforts:

• Consumer complaints of fraud dropped 72 percent
• 40 percent increase in MoneyGram's ability to identify and interrupt potentially fraudulent transactions
• For the first two months of 2011 alone, MoneyGram identified and interrupted 2,200 transactions suspected of fraud, saving its customers from losing more than $7 million

Often it has been said that it’s difficult to calculate hard ROI from MDM and related technologies. MoneyGram demonstrates the business value of these types of technologies. I hope through my blogging efforts I’ll be able to share more stories like theirs in the future.

Learn more about IBM Identity Insight solutions.

Can natural language processing help genius work like a lighthouse, with flashes of inspiration?

Our lives are filled with distractions.  There are the business distractions that come in the form of urgent emails just as we began working on the project we've been trying to complete for a week. Inevitably, personal distractions will require our attention even though we may have to sacrifice our business time and effort.

With the day-to-day toils of responsibility and the endless stream of information to which we must attend, focusing our thoughts on good ideas of significant quality is not an easy matter. Multi-tasking, despite the hype, is not necessarily desirable. We find ourselves surrounded with tasks from all sides, each item asking for a bit of input, but often distracting us from the other problems we would love to solve. We find ourselves craving more time for our own creative process.

When we do catch that rare moment of insight, it is often fleeting. It is as if great ideas seem to come from nowhere, and disappear just as quickly. The novelist Virginia Woolf summed it up best in Orlando: A Biography when she wrote that genius "resembles the lighthouse in its working, which sends one ray and then no more for a time; save that genius is much more capricious in its manifestations and may flash six or seven beams in quick succession… and then lapse into darkness for a year or for ever."

In order for us to transform our "genius" into action, we must move quickly. Fresh ideas have a way of turning sour as we second-guess ourselves and the self-criticism sets in. Just putting the thought into words is a vital first step to prove our ideas are truly innovative and worthy of our time. To validate the feasibility of our ideas, we go to our favorite search engine, or drive to the local library, and start seeking the answers for the many questions we face.

The entire process of creative thought, from birth to inception (or untimely death, as the case may be) is time-consuming, cumbersome, and may lead to various dead-ends along the way. Each step may lead to a loss of faith and the idea could evaporate under the distractions around us. In order to come to fast conclusions, it is imperative to make the process as efficient as possible.

I believe this is where the natural language processing (NLP) systems such as IBM's Watson play a major role.

We think in words, and words are an extension of our natural language. Rather than needing to translate our thoughts into a written format (as we do when we query search engines), or designing more solid definitions through intensive study, computers geared towards natural language have the ability to process our spoken questions on-the-fly, saving valuable time and effort.

Ultimately, ideas with rapid positive feedback can flourish, achieving much higher quality results and perhaps even spawning additional creative concepts that could benefit the masses.

This is not to say that tools like Watson will be the end-all, be-all reference guides for each idea that has a potential for success.  It has always been human intelligence that drives innovation, and humans will continue to be the best judge of their own insights. By keeping creativity at the highest level of language, people can focus on the concepts instead of worrying about proper computer syntax or formal mathematical logic. That's one less distraction in a world where time for creative thinking is highly valued.