I am Jeff Silverman, a Data Analytics specialist who, thru analytics, has captured terrorists, broken drug rings and even helped out commercial companies trying to save their businesses. AMA!

Are these AMAs just you and companies alike trying to cover their asses due to the recent bad press the industry has received? Because the same datasets you collect to supposedly do good can also be used for social and political manipulation and you regularly sell datasets to anyone willing to pay regardless of what they plan to do with the data.

The spin shoved down our throats reads just like your title, give up all your privacy so we can catch terrorists and other wrong doers ... if it's been used this way it should be publicised heavily when you break a case, we should hear of both the good and the bad uses of our data! Yet every time we hear about a terrorist attack or shooting you read statements from the FBI and other letter agencies like ohh yeah we knew about this guy, a even arrested him a few times/visited his house 40 times over the years... oh so you knew about the cunt but you waited for him to kill 10, 30, 50 or so people before you decided to arrest or shoot the prick!?

Show the public that your data is working and preventing these crimes becuase I'm fucking sick to the back teeth of hearing the oposite and having my privacy stripped away at every opertunity people like you get!!

To see OP's reply to this question, click here or scroll down.

Hey /u/phonetriffid, not the OP (and I want to point out that I work with Western Digital, who suggested Jeff do this AMA) but I want to point something out.

There have been a lot of abuses of big data, lord knows you hear about more every day, but there are many positive ones brewing, too. UCSF for example is working to build a database of breast cancer screenings in order to use data to develop a way to improve screening technology.. This field currently has its hands tied by government regulations regarding privacy such as HIPAA -- which in my opinion is a totally good and necessary thing, just in this case it is slowing down legitimate/positive progress.

We've written about other positive uses of big data at our site, Data Makes Possible. There are no marketing messages or company fluff on that site -- it's just articles, many of them written by WIRED Insider or data scientists like Kirk Borne.

I think every advancement or change comes with both positive uses and abuses. The world is struggling to come to grips with tech like big data and AI so we're going through that painful learning phase now. But I for one am optimistic we'll come out ahead somewhere down the line.

Edit: Fixed Kirk's surname

Thanks for this I'll take a read and generally do some research on the positive uses I'm currently just frustrated with the points I outlined.

Also you mention regulations holding a few things back. That often seems to be the case policy, ethics and religion in government holding back progression in 100s of fields. Again a frustration of mine.

I'd still like to hear from OP here as well.

Happy to weigh in. There are good eggs and bad ones when it comes to access to data and how to use it. I will tell you of the various analytic teams I have led over the years, we are very careful to adhere to EO 12333, the Intelligence Oversight executive order that limits collection on US Persons unless it meets certain legal criteria. One of those is Law Enforcement purposes, which is authorized under specific charters. A lot of times we in the analytics community cannot divulge how such tools were used to stop the bombing, otherwise we tip our hand for the next attack. I know its maddening to have to trust that people in these positions with this access are not visible to the general public, but I will tell you there is STRONG internal policing to ensure people follow the rules. Still, there have been breaches of protocol (see good vs bad eggs) so it is a dilemma....just know, that the vast majority want to do right.

Here is where the eternal dilemma of "the government knows whats best for its citizens" versus " the government is seeking whats in the best interest for its citizens" comes into play. Having worked in the federal government, i don't think this dilemma will ever be solved for the same reasons you stated.

But every event (terrorist or other) that occurs pushes us to a more surveillance state and flexes the muscles of these policies further and further to the point where we find ourselves today.

Well put. I completely agree.

please inform us how its possible that a database of people who have been screened can in ANY way change the technology used to screen people in the future,. it literally cannot. thats like saying a database of people who have had fillings done, can help develop better laser drills. Completely disparate data.

Seriously letsthink for a sec
Tech Engineer- " hmm, i need to develop a better way to screen for breast cancer, now if i only had a list of names of people who had screenings, then id know how to " insert next line please?

Not an expert in screenings, but wouldn't a the success rate..ie if the cancer came back, would help to determine the efficacy of the screening itself? So...if the screen of 1/3 of the women failed...perhaps we notify the other 2/3s?

These bullshit data-analysts are anyone who couldn't hack an actual statistics degree in college, because you know... math.

Hmmm....I took Statistics....I found my classes in Modern Physics, Nuclear Reactor design, and Fluids, Statics, and Thermodynamics to all be very heavy in math....much more than statistics. My point is that math is great, it helps solve a problem, but it is the mindset of an analyst to frame the problem and articulate what the answer MEANS and how that answer can be utilized. Just being able to talk standard deviations doesn't get you all the way there.

Would you do an analysis of your AMA? If so, to what end?

Ha.....I guess we could chart number of comments by user, Identify the user locations, and see where there is interest. You could bring in sentiment analysis, which rates how pro vs con the words used by comment to derive if the content was well received. Taking that information you build the next AMA to optimize the likes of certain regions/people to make it more effective? Or.....conversely, we could geo locate the trolls and exterminate? J/K.....all in what you want to do...

How do you feel about the all the bad press regarding Data Analytics?

Evaluating data (and deriving insight) is not meant to be good or bad, its simply a tool. For every bad guy you stop using these methods, you can also use insights to make bad choices or take advantage of people. I think it is entirely on the user's moral compass on what they apply such analysis for. If a company chooses to abuse their access to data, they should be held accountable. I do think a responsible analyst with a proper and authorized reason can make a world of difference.

How did you break out of a prison....twice?

I was the chief of intelligence at a corrections complex. The warden wanted me to conduct a vulnerability assessment on how secure the prison was. Using only data available to an observant prisoner, I figured out a route that would make it out of the prison...so I did it....once through the roof and once through a sewer. Some of my analytics were used to overload the system (ie response force) such as triggering the motion sensors in other locations (ie throw rocks at them!) to get through to the end. I will note, all of the vulnerabilities were noted and corrected and the prisoners are still secure, no escapes!

Jeff crawled to freedom through five hundred yards of shit smelling foulness I can't even imagine, or maybe I just don't want to.

Thankfully it wasn't that long...and I just proved I could get to the sewer...

Honestly, this sounds like complete bullshit.

Sorry you feel that way. It happened. All prisons conduct vulnerability assessment to determine where there softpoints are. Sometimes its to smuggle in drugs, sometimes to see if there is a cellphone.....read my article linked above.

Also, I almost made it out a third time but I was caught at the last second in the laundry cart making it out of the facility for cleaning.

Do you believe you got caught that time 'at the last second' because it was standard procedure to check the cart, or did they get lucky, or did you make a mistake?

I only ask because escaping via the laundry cart is as cliché as escaping via tied together bed sheets, so I would hope it's one of thise things they check every single time. ( for any outgoing object large enough to harbor a person, really )

They checked via technology, a listening tool that checks for heartbeats....I questioned if it was as good as they said...it was.

Could you somehow counter the frequency of your heartbeat? Or supplement your heartbeat with a similar noise so that there is no discernible rhythm?

Perhaps if I were a Jedi?

So what you’re saying is that you’re capable of collecting both visual data and quantifiable data? As someone in college and interested in this field it appears you have to have a certain mindset. It’s not just about numbers and rearranging them to make sense.

Visual data can be parsed for common patterns. Have you ever had to register online for something and it asks you look at really squiggly words or a street sign...and you have respond what it says to verify you are a human? That is actually (this isn't classified..look this up) a tool RECAPTCHA which takes all the responses and files them as potential answers to the picture in question...ie you are machine learning what that visual street sign picture is. So next time a similar picture is found, it can have a high degree of confidence of what that visual is.

As a statistics undergrad, what do you think are some topics I need to be proficient at if I want to have a career in analytics?

Complement the stats with technology skills. Your stats will help you tell your graphs what they should be doing, ie what regressions..etc, but you will need a tool to plot it and bring in large scale data sets. Understanding business (not high finance...but more like operations) will help you be able to then transfer your technical knowledge to actual insight.

Learn to program. Python is probably the way to go if you're just starting out.

And SQL?

Yes SQL....I think that is #1

On a side note. Do you know R, Python and their libraries, hadoop, sparks, etc.. what do you know?

In order of knowledge : SQL, Hadoop, Spark, R and then Python.

Hi Jeff, where does the data when researching into the crime and terrorism come from? Is it a mixture of offline and online? How do you combine the two and provide insights?

My experience in criminality and terrorism has been with working with the FBI and Armed Forces. Most of the time, we are not using a lot of public information. Intercepts, debriefs..etc are all not commonly available HOWEVER, social media does offer some insight into patterns of life, so that can be included to help paint the mosaic of where the bad guys will be.

I dabble in analytics at work, but I'd like to get more into it.

Are there any resources (software, courses, books, tutorials, etc.) that you would recommend looking into?

Yes! That's great, I think it depends on what tools you use at work for the software portion. Great entry level tools for analysis are Tableau, Qlik, and Oracle Data Visualizer. They allow you to drop in a excel file and start arranging the data to make it easy to digest and hopefully lend insight. Youtube actually has channels for most of these softwares, so watch that for tutorials.

No love for PowerBI? It's hard to beat the engine behind it.

I like Power BI....don't love it. Good engine but try to do summary data and slice amongst dimension for cross tab reporting....easier ways to do it. Its a relational DB tool....sometimes you need OLAP, or big data..etc.

I recently graduated with a degree in Public Health and plan on entering the NGO industry. I have some background in data which is desperately needed to better use/manage funds in the public health sector, but many of the locations where we work it is very difficult to get reliable data.

What tips might you have to get useful data where its not readily available and ways to easily get data without spending lots of money? Specific data points that are easy to get and useful? Specific software that is reliable? Etc...

Data that is unclean (meaning not normalized) can be pretty tricky. I actually had a similar business analytics use case where a pharmaceutical company was trying to watch where marketing money was going to in South America (certain pharma reps had petty cash to spend on clients, wine and dine..etc). There was concern of impropriety on how the money/expense were being used. We utilized an information discovery toolset (Endeca in this case) to ingest the non-normalized data (ie expense reports) and then leveraging big data capabilities within the system, we could "join" the various expenses to see how they were arrayed and who they were being spent on. The big finding was that some pharma reps were pooling their resources to influence big clients (which violated the rules...too much money to a government official). The ability to look at the disjointed funds gave insight on where they went versus where they should be going. Your NGO use case could be similar, as you want to track the money but its not always managed the same way.

What is the most underrated way people expose their privacy online?

Geo Location on pictures. When that is on, people can track where you have been.

I am a privacy professional at a firm that is really investing in data analytics as the #1 driver behind our decision making. Privacy legislation here is really behind the times, which means we really have a lot of unregulated power in big data. So, my job is quickly becoming keeping analytics based processes and decisions privacy-sensitive.

My problem is that this is not my education or experience. I need to learn a lot about data analytics/big data because this is 100% the most valuable knowledge base to have in privacy. How would you suggest a person coming from a legal background learn about the foundations of big data? I'm coming from a legal background and it seems I really need to get some CS/math/eng knowledge fast if I want to be the most valuable option in a really underserved area of privacy.

TLDR how do I learn to data

Much of the information of big data is readily available, just google it...the tough thing is filtering out the good vs the bad. I think from what you are trying to do which is to make your clients aware of their vulnerabilities, is more awareness of what is publically available, so you could definitely do that without technical training.

Hi Jeff, what would you recommend to someone trying to break into the Data Analytics field?

I mentioned in other spots, but if you are in an office role, you could co-opt a project and take an analytic approach to solve the problem. Grow your experiences that way, then when the time was right interview for a full time role and have a great resume to support your interview.

What metrics should more businesses be watching that they aren't?

Businesses usually have their core metrics down cold. A manufacturer knows how many widgets were built and how efficient their tooling machine is. I think the commonly underserved area is what I call the "Seams" how things in different areas interrelate to each other. So...when manufacturing export goes up...so does overtime, and so does attrition. So an HR question is served by an Operational answer. Those seams are very rarely reviewed as the proponent doesn't think its their concern.

If you were living in the comic book world, could you use your skills to find out the identities of superheroes and villains? Like Batman is the billionaire Bruce Wayne for instance?

Ha...the part you need to realize is that Batman is the Data Analytics guru of Gotham!

What is the most important piece of a data analytics project? Asking the right question, having a "complete" dataset, trusting the results, or something else?

Right question...and the answer drives action. If you answer the question, and the immediate response is..."well that's nice....but so what?" you asked the wrong question. Even an incomplete dataset with the RIGHT question will get you at LEAST directionally correct most of the time.

Hi there,

I just started working as an analyst and am self-training myself on R and Power BI. Do you have any other recommended training or certifications within these subjects or other topics?

Start looking into big data...Hadoop, Cloudera, HortonWorks...all the fancy named tools that look at millions of records. You have approaches to look at data, but real insight can be gained from looking at a large volume of data (and most company's are starting to invest this way, much cheaper to store data in a data reservoir/lake vice a data warehouse).

How many analytics failures have you had and what was the flaw in the model that caused it?

Countless failures. Usually its from flawed assumptions, such as formal education has a role in terrorist hierarchies (reality, it was religious education from what we saw). A lot of times you throw something against the wall, see if it sticks, rinse/repeat, until you find it.

What role do Geographic Information Systems play in your day to day analysis?

Depends on the problem set but GIS systems can play a key role in most analytic operations. The "where" aspect of an event is usually a key driver to the decision.

So I'm interested in this kind of work. I'm more interested in detecting white collar crime with these techniques, but similar ish. So you have any insights in getting started in the field? I'm gonna be graduating with a PhD soon, and want to position myself well.

I see you have done work with the FBI, which is an organization I'm Keen to work with. I have searched their job ads, but don't really see what I'm looking for. What kind of search terms would return this kind of work?

Biggest thing to look for is an Analyst Position in the Bureau...not a Field Agent.

Really interesting AMA. My question is concerning career paths. My most recent employment was as a category and retail analyst for a marketing company, but I had to take a long leave due to Multiple Sclerosis and now I am considering a return to work because my neurologist finally agreed it could go well now. However, due to the length of my leave of absence, it seems that I am unable to return to my former employer. The problem I am facing is that I also have no degree (I know, I know, hopefully that will be in the works in the future) and only a small amount of analytics experience from the logistics company I was with before the marketing company (I saw an opportunity there and created the role while originally working as a dispatcher)- between the two jobs, I had around four or five years of useable experience. I also studied for CPCA certification with coursework provided by the marketing conpany, but was unable to apply for certification before the complications from MS forced me to take leave. The timeframe to apply has obviously passed and I cannot afford independent coursework to get certified, plus I don't know if the cert without a degree would even be worthwhile in job hunting.

What would your advice be for me to get back into analytics? I was passionate about using data to solve problems and would love to go back to that field.

Thanks for doing this AMA, and thank you if you respond!

Its tough once you are out of a field to get back in. Not sure if this is something you can do, but you could volunteer to a larger non-profit organization....perhaps supporting MS? For the volunteering, you can help them with an analytics problem, as people can volunteer the three T's (time, talent, treasure)...so you are providing the talent. That way you can get some bona fides on your analytics as you pursue the next?

Given the prevalence of ideological viewpoints in today's media reporting (aka "fake news"), I feel that it is important to rely on objective data to make judgements. How do you start looking for publicly available, high quality data sets on a new topic? Any paid sources that are worth subscribing to? Who has good visualizations of this data?

The domain I am interested in is mostly related to high level national issues. For example, as soon as I hear US-China trade war, I want to see for myself - what do US-China trade export/import numbers look like? how have they changed over time?

But more local examples also exist - where does most of my city's spending go to? Where is most of the crime happening in my city?

I know this is a very broad question, but any advice you have is useful, really.

The GAO and CBO are both trustworthy and non partisan. US Census Bureau is fair....those can give some good info. Scoff if you will but the CIA World Fact Book I find trustworthy but I understand the name is off putting.

What are the most common statistical tools you use (after dealing with the low hanging fruit using graphs)? Linear Regression, ANOVA, CART, Random Forests, Logistic Regression, Cluster, Discrimination, Neural Nets...?

Linear Regression is usually the most accessible to a non- statistically minded executive....so I find that and Cluster to be easier to explain, thus easier to build upon from organizational sponsorship.

Hey Jeff,

I also enjoy analysing data as a hobby. Any advices for who is beggining to do this? (Using R and Stata right now)

Also, nowdays geting the data fets is fairly easy, but asking the right questions, filtering properly and indentifying/removing outliners is harder. Any advice?

I would start with the question and find the data that can answer it. Its a hard to stare at raw data and ascertain the question after the fact.

Outlier can be more easily dealt with....lean on statistics to determine anything 3 standard deviation's away.... however, I do think sometimes those outliers can be pretty interesting to take a microscope too.

For example at the prisons, we had inmates call volumes (how many calls, how much time) available, although not enough time to listen in to their calls (which we had...it was all legal, I should add). So, using statistics, we evaluated that people with high volumes beyond the norm (so 3 sd's) may be in distress. We had our chaplain talk to them just to check in, and one inmate was being severely bullied and two suicidal, and three had normal issues...like going through a divorce. But the stats helped us sort through the chaff to a manageable number.

Hi Mr. Silverman, I'm currently a college student that is working towards a career in the intelligence community and was wondering if your job is related to it at all? Specifically, do you do the same thing as intelligence analysts do? How can I gain gain work experience in your world? Also, every intelligence operator should have some ability to analyze and make meaning of the information they pull so what ways could I prepare myself for such work? I would say I have a very logical and detail oriented mind in which I find meaning in things with ease. I also went out of my way to find an internship with a local PI firm and have been learning basic tradecraft for intelligence gathering and whatnot.

I am also an intelligence analyst, I think data analytics is just another way to say it.

Not trying to be a recruiter, but if you really want to do it, the military got me started. I think that work is much more applicable than a PI (tradecraft and such is less what I do).

I am curious in what branch and occupational specialty you are in regard the reserves. I am currently in the Army Reserves in an analyst MOS while also attending university as a Business Information Tech major with a minor in business analytics. My end-goal is to be part of a larger company or firm as a data analyst, but still maintain my military career. How can I use both (other than the veteran preference that is usually given) to further my career in both fields based on your experiences?

Reach out to me on LinkedIn

The end result is amazing, but is most of your job sifting through boring fluff of data?

Yes it is. But, the thrill of an accurate prognosis or hypothesis is worth spending 24 hours straight looking at all of the pictures/pieces of data. Its tough if you spend all of that time and not solve the problem.

I personally thrive on the successes but I am still bothered by the Al Qaeda financier I couldn't stop...hypothetically.

I have worked for the last 2 years in an industry where we collect information for FDI and then record relevant information for our clients. I don't mind the job so much but wonder what other kind of work I can find with my skill set in Data Analytics?

Can you apply that data collection to some insights to be gained from it? If you can gain insight and push for action, that IS analytics.

Thanks for the AMA! My question is do you think Analytics could be used to identify people that are high risks for becoming mass shooters? I know it’s a matter of having the right data sets but there seems to be some patterns in Social Media, purchasing and other things like SSRIs. Not to automatically take anyone’s legally owned weapons but to trigger a deeper level of inspection by an appropriate agency.

Its a serious question and the identification of Lone Wolf's is a serious and hard to deal with problem (ask the FBI, they will agree). The best I could offer would be to detail a profile of what these individuals have in common (which is hard to do) and also ID what "gates" they would have triggered to get to their evil deed...so buy a gun, have a conflict with a coworker or student..etc. The issue is you can get a lot of false positives which interferes with freedoms, so...its the tradeoff...how much policing vs how much freedom, is profiling fair, is it biased? I think that is why we haven't solved it very well.

What advice would you give a college student who loves data analytics?

Is there any specific advice you would give to someone who wants to consult or start a business in data analytics?

Find ways to use these methods now. To plagiarize again from my upcoming article on LinkedIn .....find questions you can analyze (whether its how successfully your Fraternity is at recruiting, or your college team is at free throws) and then have your answers your provide a "SO WHAT" which should drive action...here is my snippet:

The most critical step to calibrating an analysis is to finish each insight by asking the all-important: “so what?” Challenge if the information is simply anecdotal or is there a next step or action that can be taken now that the insight is known. Finding the real answer to the “so what?” question helps drive the information to be actionable and thus intelligence and is the critical value-add piece of the entire analysis. The “so-what” factor can be used to help redefine analytic questions and prioritize the corresponding actions/answers to those questions – it drives the insights gained into action.

Hey Jeff. Do you think that going forward will there still be a future in data analytics? Will the negative perception around the use and application of analytics impact the future employment opportunities?

Analytics are deemed one of the fastest upcoming professions within the business world. Companies want to know what will happen next....so, any insight is hugely valuable. I think the public can have negative perceptions of misused info but that is not blamed on the tool. Would you blame the hammer or the contractor for building a poor deck in your backyard? I think employment opportunities in the corporate world are only going to increase. For government work...I would also echo this.

What kind of data analytics software do you use? I have experience with Tableau, wondering what else is out there!

Tableau is level 1.....basic visualization but requires a structured data set to visualize from. Other tools that do the same, Oracle Data Visualizer, Qlik...etc. Next level up is tools that you can manipulate the data set, bring in joins, dimensions...things like OBIEE, Power BI, BO...etc. Next up from that, where it gets interesting, is when the data itself needs some massaging, so that where your big data toolsets come into play, so a Hadoop infrastructure with Info Discovery Toolsets ontop to see what are questions you hadn't thought to ask (I have a softspot for Endeca).

Thank you for your contribution to international security!

As member of the same community, what do you think about using data anlytics to thwart the threat of climate induced conflicts?

I hadn't ever considered this but want to clarify what you are asking? Are you saying to identify if weather patterns has an affect on conflict? Or identifying if conflicts are caused by climate? And we mean climate as in weather, correct?

Can you use a metaphor to explain to us how data analytics can be used to capture a terrorist (for example)?

Hypothetically.... you could chart known enemy sniper locations. By evaluating the terrain using elevation maps, you could identify where the best line of sight is to certain cross roads or trafficked areas. That analysis then gives you suspect locations on where the enemy sniper would sit to make his kill. Narrow that down to most likely places with quick egress routes (they shoot, then scoot). Now you have the where...what about the when. You can chart previous snipings, time of day, visibility conditions, weather..etc....see if there is a pattern. So, if they like twilight shootings, you can have a surprise waiting for them at your suspected sites, ie, your own sniper team.

Hi Jeff, with the current opioid epidemic in the USA there are several groups/companies that are using big data to examine what methods are most effective in treating addiction, stopping trafficking, and improving overdose and rehabilitation care in the community. This has long been the practice for insurance and hospital financial practices, but now it is becoming a cornerstone of much of the healthcare industry outside of the financial side. I know that your career is dedicated to fields other than healthcare, but do you see data analysts such as yourself as a field that healthcare should incorporate more or less in the future? And if so, do you believe that you have any insights into how you would approach analysis of such a complex epidemic that you think most analysts would miss?

I ask this because you had mentioned that data analysis is simply a tool that can be used for good or bad depending on the morality of the individual. However, healthcare is a field that I think many people believe should be inherently positive in morality and inviting data analysis into the equation may open the door for ignoring morality if used to justify the wrong ends.

Interesting question. There is good and bad with introducing analytics into medical field....so I worked on a project to take medical records and derive from the DRG (Diagnosis codes) trends of common DRGs and how they interrelate. So...if you have COPD, you may also have bronchitis...Step two, ID all of those folks with either COPD or Bronchitis and then see if the doctor missed the other DRG in his/her diagnosis. So the good is, that analytics help ID things that a doctors could miss...the bad is that do we want a machine to trend/diagnose our patients, as each person is unique and should deserve personal attention. So, our suggestion was that the EMR analysis review of DRGs could supplement the doctor, but a doctor still needed to sign off after evaluation. So, we didn't save time per se, but hopefully it was tighter net to prevent a miss.

I say all of that to demonstrate, Health Care is already doing it....or trying to. Insurance is looking at it slightly differently....did they miss it, and do we pay for that miss (readmissions...). Hope that helps?

Do you use SQL Server Reporting Services(SSRS) and Power BI tool? If so, how do you rate them along with other data analytics tools?

Havent used SSRS but I am very familiar to Power BI. I think Power BI is a good entry tool, however, its relationally built which means summary trends and slicing and dicing by department or line of business are a challenge.

As someone that's looking into getting into Data Analytics would you say a double major in Marketing, and MIS, with minors in both Computer Science, and Business Anayltics would be a good start?

That's quite a course load! I think MIS, CS and Business Analytics can cross into each other a bit. Yes, I think that is a lot of good content.

Do you prefer to program your own analytics or do you use a commercial software? If one or the other, which language or software do you commonly use?

Commercial software does most of the heavy lifting. Once in awhile you can augment to find exactly what you need, but its cheaper to buy then develop.

What are your credentials?

Service Academy Graduate, US Intelligence School graduate of both basic and advance courses, Command and General Staff College Graduate, Alternative Analysis Seminar (Red Team) graduate. I am certified in traditional Analytic tools such as DCGS-A, TALON, CIDNE..etc. I am also certified in multiple Business Analytic tools such as OBIEE, OBIA, Essbase, Endeca, BDD, Crystal Reports, and Discoverer. I have a few others on both the commercial and military sides but these are top of mind. I have 15 years experience in both aspects of Analytics, with multiple tours overseas as well.

How did you take a cartel down and do they really cause a threat to us, At this time of our lives?

My particular experience was with domestic drugs not a cartel. However, I have several colleagues that work in South America in that pursuit and the overarching concern with cartels is that they destabilize a region, co-opting or even subverting civilian authority. That safe haven they have allows for the flow of drugs to places that don't want them and they only recourse is to stop the distributor. So, we can't ever cut the cancer out...just one tumor at a time, and then the next tumor pops up. The safe haven the cartel has is the issue.

What kind of data analytics software do you use for something like that? And with such sensitive things like you work with, how do you ensure clean data?

Depends on the use case and your budget for the tool. I have used full Hadoop clusters with all the bells and whistles when the government needed ita big data solution...and I have also use excel spreadsheets (not preferred) to help a fledging IT company. The key is right sizing the software/analytics application to the task at hand. You don't always need a Rolls-Royce.

As for clean data vs sensitivity. I actually think that the more sensitive the data, the cleaner it can be, as less hands have been in it, and you KNOW the source (hence the sensitivity). Something that is posted publically may not be super vetted, thus not reliable.

Thank you for everything you've done for our country!

What degree in college/ what life path would lead to being able to have a career like yours?

You are very welcome!!

I have a degree in Nuclear Engineering.....so NOT USEFUL for my current work!

That being said, much was self taught and also learned by exposure in the military. Colleges today are offering Business Analytics degrees which echo a lot of this type of work and exposure to tools. I think a curious mindset and recognizing that there are patterns in everything can help drive you in this pursuit.

Was the career swap slow and gradual or did you jump ship somewhat quickly to pursue data analytics? Did you lose your passion for engineering or did you see more options in analytics? At what age did you make your career choice?

I went to a service academy, so we all are engineers when we graduate. However, my first job was in intelligence (military), so my degree was never truly used at the time. The engineering method to solve problems is still heavily used. I have a passion for analytics from this point until now...

Thank you for your amazing service!

I would like to know what kind of trends should one follow in order to find information like you. How do you connect the dots as a general pupil?

I appreciate that, you are very welcome!

You are correct that the key is connecting everything, however, I wouldn't categorize it as a linear path. I think a better comparison is a mosaic, with lots of little pictures that that when brought together, you see the broader picture. Also, there is a LOT of trial and error. As Edison once said, you have to figure 300 ways to not make a light bulb before you get it right. SO, I may try looking at all the mosaic tiles a 100 different ways, but then when you see it arrayed correctly, you can then reuse for the next picture to get an answer more swiftly.

Hi Jeff, thanks for your service.

Do you stay in touch with any of those soldiers from your unit and do they ask you for advice now because you saved their hides then?

You are very welcome!

I would say the saving hides thing was very mutual. They did most of the heavy lifting in catching the bad guys, I just pointed them in the right direction (most of the time :) ) As for keeping in touch, absolutely, I have advised on a few areas, more on process and less on specific intelligence. An analyst assigned to a location or subject knows more about it than I do, but I can give them ideas on how to best take their knowledge and help them derive action. I liken it to a person who watches soap operas for a month versus me, who doesn't. They infinitely know the content better than I, who is the evil twin, for example. However, I may be able to provide a different perspective, such as suggesting that the evil twin plot was used in a different soap, ie here's how it will end. Hope that makes sense?

Do you even have a high school diploma, you fucking charlatan?

thru

Try responding to 300 responses....shorthand is sometimes needed....tell you what, you spell check for me, deal?

What's the biggest thing you've done as an analyst?

I received a Bronze Star for stopping 9 vehicle bombs from making it out of their manufacturing facility prior to striking the public overseas.....that's a favorite.....

How do you recommend learning insight generation? Any important books/articles that helped teach you what to look for?

Good question. I am actually finishing up an article on this myself, so it should be on linkedin in a week or so. I will plagiarize myself with a snippet on what makes something insightful:

The first step for an analyst to be successful is to understand what their role is in the production of intelligence (actionable information). The evolutionary process that transforms data into intelligence is divided into three distinct parts: a) Reception of Data. Data is the raw information that has not been processed, they are simple facts without context. Think of a stray jigsaw puzzle piece that has yet to be matched or put into the puzzle.

b) Contextualizing Data into Information. Information is the result of understanding the raw data and assembling it into a common operating picture. The puzzle piece has been placed into the correct spot within the puzzle BUT the corresponding picture has yet to be processed.

c) Determine Actions that can be made from the Information (making it Intelligence). The review of the information with its corresponding context is not where an Analyst ends, it’s where they begin! If insights can be made from the information that influences how the business/enterprise behaves, that information becomes “actionable” and thus morphs into intelligence. The puzzle pieces are in place and the picture is revealed; answering our questions to drive the course of action, or defining the next puzzle to solve.

What software do you specialize in for your work? Do you use Python, R, or something else?

We use Python and R in some instances. However, those are used for predictive algorithms.

Many times as an analyst you are not always predicting but rather conducting a post-mortem of why something happened, which doesn't require those programming languages.

I did my schooling in Political Science but have found my career has stalled out. I have recently taken more of an interest in data analytics and have done some self teaching. It seems like most jobs demand some work experience handling data which my current job doesn't at all.

Earlier this week I applied for a part time MBA, which has some opportunities to work with companies as a consultant. Would this be the right path? I'm planning on taking as many analytics classes as possible during the course.

The nice thing with analytics is you can frame most jobs into it...so you can build experience organically. For example, say you are an HR person...you could plot the degree level, the previous performance ratings and the attrition of your staff and determine if there is a correlation, perhaps a Master's degree person stays longer and performs better. If that is so, you know can prompt action, hire more of this! Or don't hire that! You just did analytics, and you can use that experience for your next interview where that is a primary focus for your job.

I think classes are ok, but real life examples sing to me when I interview people to join my consultancy.