Hah! My first data scrape


I’ve just finished Module 2 of the MOOC Data Journalism course (that I mentioned in an earlier post).

The description for this module is:

This module deals with the range of skills that journalists use to obtain data. This includes setting up alerts to regular sources of information, simple search engine techniques that can save hours of time and using laws in your own and other countries.”

And (like all the other Modules) is made up of four parts:

  1. Setting up ‘data newswires’
  2. Strategic searching – tips and tricks
  3. Introduction to scraping
  4. Data laws and sources

In Part 3, I learnt to do some basic data scraping. This, essentially, is a way of grabbing content from lists, and tables, on web sites.

We covered a few tools that make this possible. The one that did surprise me was that you can use a spreadsheet created in Google Drive.

The command is IMPORTHTML(url, query, index)

Just as a practice I used it to scrape the list of Titanic passengers from Wikipedia.

Here’s the Wikipedia link: http://en.wikipedia.org/wiki/List_of_Titanic_passengers#Survivors_and_victims

And here is the Google spreadsheet that I imported the data to: https://docs.google.com/spreadsheets/d/1g_ngM049ZgAPh25UXMmwWwMDlvGY_nD6aGbFLhBJwXo/edit?usp=sharing

It was my first scraping, and nothing fancy. Also the data does need a bit of cleaning (in one case, there was extra info in the HTML that the scraping pulled in).

Also, this functionality is not just available in Google Spreadsheet. I have read that Excel can also do this. If you know of any more, please let me know.



Today I read …"Secrets of Analytical Leaders: Insights from Information Insiders"

03eabdf45949862b8e07c5936 This is the another post in my “Today I read …” series where I aim to summarise. or recapitulate, excellent, and educational, articles that I have read. (Or in this case, a webinar that I have seen).

Previous: Today I Read … “Systems Thinking in Business Analysis: Why It Counts”

Today I followed the webinar on the IIBA site: “Secrets of Analytical Leaders: Insights from Information Insiders: Why It Counts”, presented by by Wayne Eckerson.

While it became evident later in the presentation that Eckerson was talking a slightly different language than the audience (he used the term “business analyst” to mean someone who worked with BI), Eckerson, president of BI Leader Consulting, and an “Expert Blogger” at TDWI, discusses four main topics:

Continue reading

Is being Socially Connected online really that damaging?


On the Scientific American website, the other day, I came across an excellent article that discussed some of the common grumbles that people have about being socially connected online. (“I don’t care what you had for breakfast.”, “How about talking to some real people!”, etc.)

The article was written by Dr. Elizabeth L. Cohen, an Assistant Professor of Communication Studies at West Virginia University, and Dr. Rachel Kowert, an Associate Researcher in the Department of Communication at the University of Munster.

The article was in reaction to a video that talked about the damage that being socially connected has.

I really liked the article because I also feel that being socially connected is not bad. Or wrong. It is just another way of being social.

I have been in touch with Dr Cohen, and Dr Kowert, asking them if I could reproduce their article in my blog. They have graciously agreed…

In his viral video, Look Up, Gary Turk emotionally appeals to viewers to unplug from their social media (just as soon as they finish watching the video, of course). Cell phones, online games and social network sites are all depicted as distracting us from intimate human contact and a cause of loneliness.

The video, which has racked up more than 37 million views on YouTube, appears to have struck a chord with many people feeling disillusioned with being constantly connected. But before you get all sentimental and throw away a perfectly good iPhone in a pool of your tears, let’s take a step back for a minute.

Current communication and psychology research paints a much more complicated picture of how these technologies affect our social well-being. A full refutation of all the arguments implied by the Look Up video would be worthy of a dissertation, but inappropriate for the scope of this blog. Instead we’ve picked seven claims to compare against current research.

 Claim 1: We are connected to lots of friends on social media, but we don’t really know each other.

Truth: While it’s true enough that we can’t know everybody that we are digitally connected to intimately, we don’t think that’s the point. Social technology plays an important role in helping to maintain our strong-tie relationships with people we already know. Social network sites also enhance our weak-tie connections and raise our social capital, which can lead to a number of positive outcomes such as improved health and civic engagement.

Claim 2: We share frivolous bits of ourselves on social media, but leave out anything meaningful.

Truth: This is the classic, nobody-cares-about-what-you-had-for-breakfast complaint. But why should you care? Because what we had for breakfast is valuable, potentially meaningful social information. One status update can be frivolous on its own, but over time, these seemingly insignificant bits of information about what people are doing, what they like and where they are can coalesce into a sense of others’ presence, providing a peripheral but intimate awareness of that person.

What’s more, posting status updates on social media isn’t just valuable for followers, it’s also good for the posters. Experimental evidence suggests that just the act of leaving a status update can make people feel less lonely, presumably because posting reminds us that we are part of a larger network.

Claim 3: The community, companionship and sense of inclusion provided by social media are illusions.

Truth: The community companionship and sense of inclusion provided by social media are real. recent study found that people who use social network sites to interact with existing friends felt a greater sense of connection to them and reported a greater sense of belonging than those who don’t. Our own research also provides preliminary evidence that simply monitoring other people’s activity on social media can help fulfill basic human needs for belonging.

Claim 4: Online games are socially isolating and not a worthwhile way to spend time.

Truth: Our research suggests that online game players are often stereotyped as being anti-social, reclusive and isolated, but online gaming is actually highly social, requiring players to interact with, coordinate, lead and compete against hundreds of other players in a shared space. In many games, socializing is actually rewarded because player coordination eases the difficulty of in-game tasks. Research also indicates that gaming can support pre-existing relationships and help people develop new relationships.

Claim 5: Kids don’t play outside any more because they are always on their technologies.

Truth: Nobody can deny that digital games can be more fun to play than hopscotch at the park. But is staying indoors to play really so bad? These days, digital games promote exercise and social interaction with others.

But social technology might not have anything to do with kids staying inside. In her new book, It’s Complicated, danah boyd discusses the influence of technology on teens and “tweens.” Her anthropological study suggests that the real culprit behind the empty playgrounds after school has more to do with parent culture than it does teen culture. Over-scheduled and over-protected children don’t have much time for free play outside. In fact, connecting through social media is sometimes the only way kids can connect with their friends outside of teacher and parent supervision these days.

Claim 6: It’s become abnormal to talk to strangers on commuter trains because people are too involved with their personal technologies.

Truth: For those of you who can remember riding a train, bus or elevator when people didn’t have mobile devices, ask yourself how often you remember looking up, making eye contact with strangers and talking to them. The truth is, it’s always been taboo to talk to strangers, and as long as there have been trains, we’ve found things to look at besides other people.

Claim 7: If you look down, you could miss the love of your life.

Truth: Perhaps. But if you don’t also look down at your online dating profile you can also miss the love of your life.

Of course, Look Up didn’t get everything wrong. Never looking up can be both rude and dangerous. Because our technologies develop more quickly than we do, we definitely have some catching up to do on developing social etiquette and public policies that will keep us courteous and safe.

Still, suffice it to say that we think the video stretched the truth about how damaging media use is for our relationships. Turk’s fears are nothing new, though. Virtually every technology innovation has been met with some trepidation about how it will affect our social well being. Even the Walkman was accused of making listeners more narcissistic and detached from other people. But in the long run, it was nothing to be frightened of. We’re willing to bet that your iPhone is probably safe too.

What do you think of that? Do you agree?  Don’t agree? What are your thoughts?


  • Tackling the Gamer Identity Crisis
  • The Effect of Gaming on Social Interactions
  • Look Up Exaggerates Damages of Social Media

Journalism with Data


If you browse through the posts in this blog, you’ll see that there are several that are related to “telling a story”, “using pictures to present data, and similar:

Because I want to be able to present data graphically, in a proper way, I have started an online course titled: “Doing Journalism With Data: First Steps, Skills and Tools“.

It’s a 5-module online (MOOC) introductory course  that “gives you the essential concepts, techniques and skills to effectively work with data and produce compelling data stories under tight deadlines.

Awfully exciting stuff! It’s actually being taught by 5 tutors (one for each module) from Britain, America, and France. Here are the five modules:

Module 1 – Data journalism in the newsroom
Module 2 – Finding data to support stories
Module 3 – Finding story ideas with data analysis
Module 4 – Dealing with messy data
Module 5 – Telling stories with visualisation

You can read more about the course here.

I’ve just started module 1 (along with 21,280 other students), and I’m keen to work my way through the rest of the modules.

At the end, I’ll give an idea what I thought of the course along with any real gems that I got out of it.



  • Journalism Course
  • Launching a MOOC for data journalism
  • Top 10 skills new journalists should have

Today I Read … "Systems Thinking in Business Analysis: Why It Counts"

System This is the another post in my “Today I read …” series where I aim to summarise. or recapitulate, excellent, and educational, articles that I have read

Previous: How to Use Enterprise Architecture to Deliver the Right Solution

Today I read an article on the Business Analysts Learnings site: “Systems Thinking in Business Analysis: Why It Counts“, authored by Stephanie Famuyide.

The title of the post piqued my interest. I didn’t quite understand what this “system thinking” was about, but in the interests of continual learning, I read it with enthusiasm.

In her opening paragraph Stephanie points out that “Very rarely do most people sit down to assess how they think.” She points out that, as Business Analysts, we’ll often encounter situation where the normal rules of logic are not helping.

This is, apparently, where systems thinking can shine. So what is systems thinking?

Systems thinking can be described as “trying to understand reality by examining the relationship amongst the parts of the whole and the relationship of the parts to the whole instead of examining only the parts

Yes…that’s it. System thinking means rather than thinking about the individual parts, think about how they relate to each other, and how they relate to the whole.

Wikipedia makes a valuable contribution:

Systems thinking has been applied to problem solving, by viewing “problems” as parts of an overall system, rather than reacting to specific parts, outcomes or events and potentially contributing to further development of unintended consequences

Stephanie gives examples of how system thinking can be useful in the world of business analyst.


As mentioned – I started off with a lot of interest. When I read what systems thinking actually was, i was a bit underwhelmed. I did some further research on the subject, but was not able to get more excited.

Still, it’s a good reminder to not get entrenched in one particular way of thinking about things. In one of the other articles I read (Overview of Systems Thinking, Dan Aronson), states that the word “analysis” comes from the root meaning “to break into constituent parts”. System thinking is the opposite of this.

The original article can be read on the Business Analyst Learnings site here: http://businessanalystlearnings.com/blog/2013/5/29/systems-thinking-in-business-analysis-why-it-counts

Note: Stephanie Famuyide has written several other articles that all look like they offer value to business analysis.

If you like this post, feel free to share. If you have comments that you’d like to make, please go ahead and use the comment box below.

Thanks for reading.

Further reading
  • Systems thinking (2014, March 31). In Wikipedia, The Free Encyclopedia.
  • Systems Thinking (Links  to Systems Thinking related information)
  • Overview of Systems Thinking, Dan Aronson
Comments from LinkedIn

I posted this on a few of the LinkedIn BA groups and this created some interesting comments. I have listed these below:

BA Forum

Meg Little – Business Analyst/Project Manager at Sabre Holdings

Thanks for sharing. I had not visualized ‘System Thinking’ as the term. Just that in many companies (mine included); it is more common to analyze the PART as the project or requirement instead of the SYSTEM it is integrated into.

Manny Luna – Sr Business Analyst – Consultant

I am glad to see that finally some one has actually come to the conclusion of my many suggestions in several posting, which is: A BA that only focus in the “Functional Side” of requirements is leaving a lot of room for failure, as a project needs to be seen beyond the functional aspect, it must include other groups that directly or indirectly are touched by the needs of the client, thus the knowledge and experience in those other parts is critical to the success of the project. Having Technical as well as Functional experience allows any BA with the ability to discover things that most of the time are not discovered by simple logic, but will make sense when we see how those other parts being impacted will provide a more complete solution to the client’s needs. 

BA need to further their knowledge and experience to include the other parts, or at least surround themselves with those that can provide the necessary guidance to see what a single pair of eyes can see.

Jo Keown – Business Analyst at National Heavy Vehicle Regulator (NHVR)

When I read your article Mark, I was a little underwhelmed. Not by the article, but that it wasn’t really a revelation. You see, I’ve been under the misapprehension that this was called ‘holistic thinking’, which I’ve been doing all along. Although, I must hasten to add, you are correct in saying that it isn’t always embraced by others as some think my approach is a little over the top. However, I believe, emphatically, that you can’t address a subsystem without looking at the whole picture and viewing it in it’s context, particularly if there are dependant inputs & outputs. A further point worth noting, my holistic/systems approach has never lead to a failed project, conversely, I believe it has been a large component of the success of these.

If, however, there are time/budget/political constraints that prevent me from taking a holistic/systems view, then I always call out additional risks and ensure that these are noted and understood by management and SME’s.

Larry Felton – Senior Business Analyst, Technology Partners

Systems thinking deals with archetypes — specific forms of interactions that will lead to specific results. The 5th discipline by senge is an excellent source. It also provides excellent methods for gathering requirements.
Chuck H.- Experienced Business Analyst

I enjoyed your article. Sadly in my work place, I’ve been accused of thinking outside the box too much and am penalized for it. I receive comments such as “why are you working on that; your part is a minor subset of *that* and that’s where you need to be concentrating. I hate being the “I told you so” voice, but when problems go wrong down the road it usually happened in the area I was looking at in the first place and then was told not to.

IIBA (International Institute of Business Analysis)

Duane Banks – Holistic Analysis and Solutioning 

Though business analysis is a practice that will benefit from virtually *any* new learning, I’ve come to the conclusion that systems thinking is not uniquely applicable to business analysis.

In fact, I equate the two, with an essential difference. Whereas systems thinking is the practice of “understanding” Complexity (actually, *engaging* Complexity), business analysis seeks to understand Complicated matters.

But note that I observe a particular definitions of Complex and Complicated, nicely described at the following…


Putcha Narasimham – Consultant and Coach, BA, RE & Solution Design

What Duane says may be right but I have seen a direct benefit of applying Systems Thinking in the following manner. I hope it qualifies as ST.
This is interesting but I find that two principles of General Systems Theory are NOT spelt out. I am unable to find the source of what I read but I find the following are very useful for discovering hidden factors and gaining insights.

1 A system is made up of interrelated and interacting sub-systems (parts),
2 A system is a sub-system of a larger system.

By themselves they are NOT very useful. Their implications ALSO must be elaborated, appreciated and applied.

I do not recall reading the following anywhere but I do not claim originality for the same.

Implication and application of 1:
It is sufficient to identify and separate (not necessarily physically) the largest sub-systems of a system. After that a study and understanding of how the specific subsystem interfaces with other subsystems is sufficient. The sub-subsystems of each subsystem need not be probed.

This allows minimal “digging into details” for SUBSTANTIAL understanding and tacking of the system.
Implication and application of 2:
This is the principle that reveals the external factors of the system that are essential for the operation of the system under discussion.
Through this we discover how gravity and friction are necessary for a car to gain traction and why the car does not run underwater or on moon.
This also explains why ecommerce did NOT work or inspire a general buyer to buy on the net, till laws and assurances came forth to define the modes of operation on the Internet and their correspondence with general laws of commerce.

While I ask questions and explain only after the students do their bit of exploration, there are great rewards of student’s independent thinking in many fields.

One such is application of GST 2 to the ISO 9000:2005 definition of System. The student pointed out that it DOES NOT acknowledge that the system under consideration “is a part of a larger system”.

My students are NOT outstanding but they have the independent ability to examine and EVEN CORRECT ISO Definitions which are elaborately formulated, tested and validated by world class experts.


  • Your Business Analyst Resume: What to Include
  • How do I become a business analyst?

Why I hate CAPTCHA (or "I am a Human!")

I think that most people have, at one point, come across CAPTCHA (or similar), that small box that contains a distorted word that you have to type correctly.

Kill Captchas

This is done to prevent bots, or automated software from logging into sites, or filling in forms, etc.

What really bugs me about this approach (from a User Experience point of view) is “Why do I have to prove that I am human?! It’s putting the onus on me, as a visitor. The system should be putting the demand for proof on the bots!

Here are two excellent articles that cover exactly what I am referring to:

  • Time to Kill Off Captchas (Scientific American)

  • Beyond CAPTCHA: No Bots Allowed!

The Scientific American article discusses how, whenever there’s a problem in the modern world, we counter it by building barriers. Barriers that provide more inconvenience to genuine users, than for the “bad guys”.

Beyond CAPTCHA: No Bots Allowed!” goes into the problems that are encountered with CAPTCHA. How the distorted text is so hard to read, etc. It carries on describing alternatives that can be put in place. It was, however, in its conclusion that the author showed that he understood my pain, with this sentence:

Don’t make users take responsibility for our (site owners) problems.

This followed up with the succinct:

Bots, and the damage they cause, are not the fault or responsibility of individual users, and it’s totally unfair to expect them to take the responsibility. They’re not the fault of site owners either, but like it or not they are our responsibility — it’s we who suffer from them, we who benefit from their eradication, and therefore we who should shoulder the burden. And using interactive authentication systems such as CAPTCHA effectively passes the buck from us to our users.


Kapow!! There it is…don’t make the problem with bots, the responsibility of the users!

I agree totally! Do you?

(What are your thoughts on CAPTCHA?)


  • Street View and reCAPTCHA technology just got smarter
  • A Gaming Replacement for Those Annoying CAPTCHAs
  • How Captcha Turned You Into a Data Entry Robot
  • The ‘CAPTCHA’
  • Apparently reCAPTCHA has Digitized All the Books
  • I hate captchas!

My short rekindled love affair with Microsoft Paint


Yes, I am not ashamed to say it, but I had a short-lived affair with Microsoft Paint.


My first love

Back in the heady days of Windows 3.1 (if you can remember it, you were never there), I was a big user of Paint (or Paintbrush, as it was called then.) It was cool to be able to create works-of-art.


Love Lost

As I started finding my way in the world, I started using Paintbrush less, and less. I was never a graphics designer, and didn’t really need to use such tools. At most, I needed to be able to take screen shots, and manipulate these if necessary (things like trimming unwanted parts out, slicing sections out of the middle, adding text, etc.). For this I found Snagit to be an excellent tool.

Snagit is my door


New Worlds

Then I moved into the world of Business Analysis, and wire-frames, and mock-ups. If an existing system needed changes to the user interface, it was often necessary to be able to demonstrate what the interface would look like once the changes had been incorporated.


Love, rekindled

I had started a new position at a financial company, and my beloved Snagit was no longer available. I was required to document web site/application changes, and what did I have available…Paintbrush, (or as it was now know, Paint). “How can I work like this!?” I screamed to myself.



Having no choice, I started using this infantile tool. And, you know what? I was amazed at what I could do. Once I got the hang of it, I could easily paste in a screen shot, trim out unwanted elements, move elements around, even replace text. (This last one was not straight-forward – it involved getting the new text to look like it was the same as the old.) I quickly became a MS Paint advocate.


It didn’t last long

This adding the next text bit, however, got me frustrated. it was clumsy, and required a lot of fart-arsing around.

That’s when I discovered what I could do with a browser’s Developer tools …

…to be continued


Today I read …"How to Use Enterprise Architecture to Deliver the Right Solution"

Enterprise Architecture This is the another post in my “Today I read …” series where I aim to summarise. or recapitulate, excellent, and educational, articles that I have read
Previous: BA Practices in a Virtual World

Today I read an excellent article on the IIBA site: “How to Use Enterprise Architecture to Deliver the Right Solution“, authored by Sergio Luis Conte.

For me, this was an excellent article.One that really helped me get a better understanding of Enterprise Architecture, especially from a Business Analysts point-of-view.
Sergio pulls relevant information from the EABOK (Enterprise Architecture Body of Knowledge), the BABOK (Business Analysis Body of Knowledge), along with other relevant sources, to detail:

  • What Enterprise Architecture is,
  • Why it should be used,
  • When it should be used, and
  • How it should be used.

The “What”

Sergio provides a quote from Gartner to answer this one:

a discipline for pro-actively and holistically leading enterprise responses to disruptive forces by identifying and analyzing the execution of change toward desired business vision and outcomes. EA delivers value by presenting business and IT leaders with signature-ready recommendations for adjusting policies and projects to achieve target business outcomes that capitalize on relevant business disruptions. EA is used to steer decision making toward the evolution of the future state architecture. (Gartner Group 2013)

He goes further by describing how Enterprise Architecture consists of several independent, but cohesive architectures: Business Architecture (BA), Application Architecture (AA), Technology Architecture (TA), Security Architecture (SA), and Information Architecture (IA).

The “Why”

Sergio explains that Enterprise Architecture is a way of thinking about the Business from a system management theory perspective. He also ties this nicely in with information presented in the BABOK (under competencies).

The “When”

Enterprise Architecture is used when a business needs to transform itself – when a desired future state is recognised.To identify the gap between the current state, and this future state, a gap analysis is performed, and appropriate steps are taken to make the necessary transformations,

Image “In search of problem situation to solve” – Sergio Luis Conte (IIBA)

This is a repeating cycle. Businesses attempt to adapt to an ever changing environment.

The “How”

For the “How”,Sergio mentions that there are several models available for working with Enterprise Architecture. The one he concentrates on, though, is “The McKinsey 7S model” that focuses on, and analyses, seven elements – strategy, structure, systems, staff, skills, and shared values.

Sergio explains each of these seven elements in further detail, including listing references for further reading.


All-in-all, a great article that helped me lot, and gave me enough information for further reading.

The link to the full article is: https://www.iiba.org/News-Events/Best-Practices-for-Better-Business-Analysis/BP4BBA/2013/enterprise-architecture-to-deliver-right-solution.aspx

If you like this post, feel free to share. If you have comments that you’d like to make, please go ahead and use the comment box below.

Thanks for reading.

BA Practices in a Virtual World

Virtual Working This is the another post in my “Today I read …” series where I aim to summarise. or recapitulate, excellent, and educational, articles that I have read
Previous: As a BA, be aware of your motives

Today I read the transcript, and viewed the slide deck, from one of the IIBA Spotlight Series: BA Practices in a Virtual World (from May 2013). I was particularly interested in this,as I am a strong believer that projects can be completed by groups, and individuals, that are located in geographical disparate locations. The webinar was presented by Larry Simon of the Inflection Group.

(Note – this webinar is archived on the IIBA site, but is only available to IIBA members.)

The webinar, promised that I would learn the following:

  • How Facilitation has Changed
  • Building Rapport Virtually
  • Managing Participation
  • Tools for Virtual Teams
  • Demo: Powernoodle for Virtual Facilitation

This was promising.

I read through the slides and then went through the webinar transcript… After the usual introductions, etc, Larry pointed out, through the use of an example (he use to lead a facilitated centre, or an accelerated solutions environment) that the standard practice was to get everyone in the same room, and to hash things out, with copious use of whiteboards, and “group sessions”.
This would continue until a solution had been reached, or an agreement on what the problem was, what the requirements are, etc. There was an expectation that the classic Form/Storm/Norm/Perform would take place.
He then goes on to highlight the fact that, often, getting everyone in the same room, is not possible. This may be because of different geographical locations, or the fact that people work from home, or for any other myriad of reasons. (“Your office is where you are.”)

The presenter describes several handy techniques, and tools that can be used when holding a “virtual workshop”. Handy things that we should all write on a piece of paper and keep in our pocket for reference. Things such as being considerate when talking, building rapport by disclosing something about ourselves that the other person didn’t know, or mirroring others (without mocking).  Finding out as much as you can about the other attendees is also a good tip (but the presenter warns that there is a fine line between being interested in a person, and stalking them.)

Taking notes during the session is also a recommended practice. Recording the sessions is also a “really good suggestion”. I won’t describe the other incredibly useful gems that get mentioned.

Then Larry describes several tools that can be used for virtual meetings. There are tools that allow for the sharing of screens,or for sharing files, and documents (anyone everheard of Google Docs?). This culminates in a demo of Powernoodle, an online collaboration tool which, actually,offers some great functionality.

There are quite a few good questions asked my the attendees of the webinar, but these were not answered in a satisfactory way (I felt).

Expecting More

All in all, I was expecting much more from this webinar. I have seen large enterprise projects work where the stakeholders and the implementation team, were all spread across multiple cities, countries,and continents, where English was not everyone’s native language.

I felt the advice, and information, that the presenter gave was a bit thin. It did not have a lot of depth. Having said that, I understand that the field of working with disparate teams is something that cannot be given justice in an hour-long webinar.

In fact, this topic is something that I have been,and will continue to, delve into more deeply (including Agile and remote teams). I’ll keep you updated.

The webinar can be viewed on the IIBA site (members only).

A PDF of the slide deck is available on the IIBA site here, and the transcript can be downloaded here. (Again – members only)

If you like this post, feel free to share. If you have comments that you’d like to make, please go ahead and use the comment box below. Cheers  

Funny … teleconferences in Real Life

Having attended several teleconferences, I can really relate to the video below.

If you want some more chortles, click here.