SCRUM Principles

Background

---

Attended the SCRUM Workshop, in short, it is a must for all developers/Architects/PM/Program Manager/Tester/BA.

SCRUM Values

---

In SCRUM, we value: -

• Individuals and interactions over processes and tools
• Completed functionality over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

How SCRUM Helps?

---

1. Works Transparency - Everything is transparent to the corresponding stake holder, customers and even to the team. Burn Down Chart, Team Velocity.
2. Self Empowered and Organized - SCRUM Team work as a whole to estimate and agree on deliverable achievable.
3. Collaboration - The transparency of the progress, status and facts allows the SCRUM team to collaborate and interact more to the stake holders, customers on change plan without over optimistically committing to the request too easily.
4. Recognize the needs to change - The only thing remain unchanged in this world is the change. It recognizes it and put focus on what is best known currently and do it while anticipate future changes is needed.
5. Incremental Release - Encourage frequent release with each release being workable deliverable as a whole, in a smaller but manageable incremental scope.
6. More and frequent planning - SCRUM does more planning, Product Road Map, Release Plan, Sprint Plan. However, only the highest priority which needs immediate work gets the detail plan, tasks, estimation and etc.
7. More Meeting - Daily SCRUM meeting. Daily SCRUM meeting on What is done, what is not, what is on your way. 
8. More Frequent Reviews - Sprint Reviews at the end of each Sprint on what is completed and what is not by adhering to Definition of Done.
9. Sprint Retrospective Meeting - After the sprint reviews with Product Owner excluded. Discussed on what's good and bad as project is still ongoing. To know the problem earlier, learn and improve as a continuous effort and avoid repeating the same mistake within the coming Sprint.
10. Product Owner - Single person to have final authority representing the customer's interest in backlog prioritization and requirements questions. This ensure no missing link or tear down at communication.
11. SCRUM Master - As a facilitator to work with the product owner and the team to orchestrate the SCRUM ceremony.

Untrue SCRUM/Agile Myth

---

1. SCRUM does not do documents or reporting for communication!
2. SCRUM is only focus on the worker (developers) not the business!
3. SCRUM does not need a Leader!
4. SCRUM makes impossible turns possible!
5. SCRUM prevents problems and it is a "silver bullet"!
6. SCRUM is self organized and self empowered, so, "Do whatever you like!"
7. Agile Fragile! No planning and architecture needed.
8. Agile is just an iterative and incremental process.
9. Adopting Agile means betraying PMI.
10. Agile means no/avoid commitment.

Reality to Untrue SCRUM/Agile Myth

---

Reality to Myth #1:
In SCRUM, we emphasize on automated and integrated workspace. We produce artifacts such as: -

• Product/Release/Sprint Back Log
• Product/Release/Sprint Burn Down Chart
• Team Velocity Matrix
• Estimation

These artifacts are meant to be showing to everyone, where we are, how are we doing and how far we are from the sprint/release/product goal. Product backlog with each itemized entry prioritized showing the road-map of  a given product and the planning of getting it done accordingly.

Reality to Myth #2: -
SCRUM is highly business value driven approach. It focus on business values to the customer than the scope or what customer "think" they want. It ensure the customer is not buying just a set of feature scopes, but the business values to the customer, sooner and earlier. Often, we see projects delivered all the scopes their respective customers wanted but the customer isn't happy in the end because it does not fulfill the business need. This is what SCRUM is trying to avoid.

Reality to Myth #3: -
In SCRUM, we are changing the game plan. Managers are too used to Directed management style. These managers need followers. They ensure the best arrangement of work to candidate who they think best working with. Often, their best people usually does not work best under such "arrangement". Think again, the best people who does well in their work are already put on to manage role, so is there anymore best people? So, in SCRUM, we coach, mentor and lead by example. We value facilitator management than directed management. This requires a capable leader with great leadership skills and interpersonal skills to make possible.

Reality to Myth #4: -
In SCRUM, we value transparency and we focus on start delivering sooner with multiple release that ships incremental of workable solution subsets. This usually help to discover and manage the impossible sooner and allow the customers to justify their decision quicker and change of game plan earlier. Hence it helps to see the fails sooner to avoid greater lost because the nature of identifying risks/impossibles sooner, the customers often can then make justifiable decision to counter the issues earlier and has higher success rate than those that can't help in identifying risk earlier.

Reality to Myth #5: -
In SCRUM, no way you can prevent problems. However, its emphasize of being transparency will let the problems to surface earlier than one would expect. These surfaced problem are Impediments in SCRUM which requires the SCRUM master with great skills to help facilitating with the team. This impediment may be impacting the productivity of the team but SCRUM allows it to be surfaced sooner and dealt with earlier and quicker. However, to some extent, people often associate this nature as a SCRUM problem itself.

Reality to Myth #6: -
In SCRUM, we need a highly discipline members that stay focus and continue productivity with consistent pace for predictable estimation. SCRUM team works closely achieving Goals and Expectation put up front by Product Owner who represents the Customers with Customers interests and expectations. SCRUM Master as a facilitator is trying to collaborate between Product Owner and the team and orchestrating the each SCRUM ceremony. Everyone has their role and game plan and this discipline must be closely adhered to.

Reality to Myth #7: -
In SCRUM, we do more planning. Product/Release/Sprint Planning. We do more risk management. Product/Release/Sprint Burn Down. We do more reviews and we do more retrospective meeting (postmortem) than anyone else, however this are done interactively within each sprint and making us so efficient in dealing with problems because we see problems sooner, efficient in learning because we see our weakness earlier, and etc. In SCRUM we need architecture to work best, however, we avoid architecture details being too detail too soon because we recognize the only thing remain unchanged is "change".

Reality to Myth #8: -
In SCRUM, we measure our productivity velocity based on facts (the earlier Sprints' velocity from the burn down) and commit to tasks within our capability. We aimed to deliver potentially shippable solution with a current subsets of product features that delivered business values where our customers need it NOW, not just want. It is not as simple as incremental, but it has a lot to do with customers' business plans and strategy and delivering the needed business values.

Reality to Myth #9: -
Wrong. We see there are many rooms for Agile PM. While PMI emphasize on: -

• Project Integration Management
• Project Scope Management
• Project Time Management
• Project Cost Management
• Project Quality Management
• Project Human Resource Management
• Project Communications Management
• Project Risk Management
• Project Procurement Management

In SCRUM/Agile, we adopts lean process, thinking and techniques and we are just focusing on facilitating and empowering the team to help elevate, surface and addressing these issues much sooner and quicker. This would help a qualified project manager to deal/elevate/stay transparent with the risks in the interests of customers, stake holders and the company. Please visit Jesse Fewell's Agile PM.

Reality to Myth #10: -
In SCRUM, we commit to deliver the agreed task items in sprint planning that was put forward together with Product Owner, SCRUM Master and the team. We commit to stay focus and deliver potentially shippable working incremental subsets of solution features. We valued transparency and we work well with fixed cost fixed schedule projects and we collaborate with the customers on business values needed than the scope wanted. Estimates derives from the past velocity remains "estimates" and in short we practiced Agile Principle by working on estimation with commitment and remain transparent, at the same time willing to adapt to the change by encouraging collaboration with customers based on facts collected from the SCRUM artifacts.

Be careful with Linq within Exception Handling Block

Exception handling try {} catch {} finally {} in .NET always guarantee the finally clause will be executed. However, deferred execution of Linq prove to be able to make a difference, once again thanks to the side effects of linq like I've discussed in the past.

Consider the code block below: -

        protected override IEnumerable<byte[]>  DoReadAsRaw()
        {
         try
         {
          return _settings
           .FieldInfos
           .Select(item => _innerReader.ReadBytes(_innerReader.ReadInt32()));
         }
         catch (EndOfStreamException)
         {
          return null;
         }
        }

This is to read out an Enumerable of byte array. I have another code block below that will call into this method and utilizing the Enumerable of byte array as such: -

         var rawData = DoReadAsRaw();

         return rawData == null ? null : new Record<byte[]>(_settings.FieldInfos, rawData);

Within the Record Constructor, I have the codeblock to iterate thru the enumerable byte array as such: -

        public Record(IEnumerable<FieldInfo> fieldInfos, IEnumerable<T> values)
        {
            if (fieldInfos == null)
            {
                throw new ArgumentNullException("fieldInfos");
            }

            if (values == null)
            {
                throw new ArgumentNullException("values");
            }

            var fieldInfoArray = fieldInfos.ToArray();
            var valueArray = values.ToArray();

            // checks the length of field and data to ensure their length is matching.
            if (fieldInfoArray.Length != valueArray.Length)
            {
                var message = string.Format("Fields and values count mistmatched. Expected {0} fields but was {1}.",
                                            fieldInfoArray.Length, valueArray.Length);
                throw new IndexOutOfRangeException(message);
            }

            _fields = new T[fieldInfoArray.Length];
            _itemNameIndexMap = new Dictionary<string, int>();

            for (var i = 0; i < fieldInfoArray.Length; i++)
            {
                var currentFieldInfo = fieldInfoArray[i];

                _fields[i] = valueArray[i];
                _itemNameIndexMap.Add(currentFieldInfo.Name, i);

                if (!currentFieldInfo.IsKeyField) continue;

                IdentifierFieldIndex = i;
                IdentifierFieldName = currentFieldInfo.Name;
            }
        }

Guess what when unit testing?
EndOfStreamException was not handled!

the past experience of Linq's side effects helps me to quickly identifying that the culprit was the absent of ToArray() or ToList() call right after the linq statement within the try-catch block.

Soon as ToArray() is added into the code, it works beautifully and passed the unit test.

So, best practise for Linq Statement within the try catch block I would recommend is: -

If you are returning anything to the caller after the try catch block, ensure you always force create real collection by calling to ToArray() or ToList().

Thank you.

Test Driven Approach - Positive Impact for Developers - Conclusion

Going thru this lengthy of workshop series on TDD, I hoped I have covered enough here on positives impact from TDD approach. It takes a developer to practice it to see its worth.

Often in software development, we fall into traps like: -
Lengthy requirement gathering period.
Lengthy conceptualizing and creating over detail blueprint (architecture).
Over design/engineering a software component at early stages.
Lost focus on interactivity between requirement/conceptual model/design phases due to lengthy period.
Less integrability due to disconnected phases.

All these can be nicefully addressed with Agile and TDD.
Agile and TDD are just complementing each other and bringing developers closer to users and hence business.

Final question, why would one insert a break point at several area to debug when a defect was raised? Write a unit test to automate that checks please. Thank you.

Test Driven Approach - Positive Impact for Developers - Workshop IV

Now we need to implement the Calculate method within Core.Calculator.

So, there are many ways this can be done and whether this method is developed to its best code is not the interest of the workshop. Here is my sample: -

using System;
using System.Collections.Generic;
using System.Linq;

namespace Calculator.Core
{
    public class Calculator
    {
        private const string Add = "+";
        private const string Sub = "-";
        private const string Div = "/";
        private const string Mul = "*";
        private const string Sqrt = "sqrt";
        private const string Percent = "%";
        private const string Reciproc = "reciproc";
        private const string Pow = "pow";

        private readonly string[] _keywords = new []{ Add, Sub, Div, Mul, Sqrt, Percent, Reciproc, Pow, "(", ")", ","};

        private readonly string[] _operatorKeywords = new[] { Add, Sub, Div, Mul, Sqrt, Percent, Reciproc, Pow };

        private static readonly Dictionary<string, Math.MathOperatorHandler> Operators;

        static Calculator()
        {
            Operators = new Dictionary<string, Math.MathOperatorHandler>
                             {
                                 {Add, Math.Add},
                                 {Reciproc, Math.Reciproc},
                                 {Pow, Math.Pow},
                                 {Sub, Math.Subtract},
                                 {Div, Math.Divide},
                                 {Mul, Math.Multiply},
                                 {Sqrt, Math.SquareRoot},
                                 {Percent, Math.Percent}
                             };
        }

        public string Calculate(string formula)
        {
            double temp;
            if (double.TryParse(formula, out temp))
            {
                return formula;
            }
            var values = formula.Split(_keywords, StringSplitOptions.RemoveEmptyEntries);
            if (values.Length == 0)
            {
                var msg = string.Format("No values to be calculated. Values length is zero for formula {0}.", formula);
                throw new UnexpectedCalculationException(msg);
            }

            var subPart = formula.TrimStart(values[0].ToCharArray());
            var opCode = _operatorKeywords.FirstOrDefault(subPart.StartsWith);

            if (string.IsNullOrEmpty(opCode))
            {
                var msg = string.Format("No calculation operator found in formula {0}.", formula);
                throw new UnexpectedCalculationException(msg);
            }

            Math.MathOperatorHandler mo;

            // this is only possible when there is a %.
            if (subPart.EndsWith(Percent))
            {
                if (!Operators.TryGetValue(Percent, out mo))
                {
                    var msg = string.Format("Cannot resolve calculation operator of {0}.", Percent);
                    throw new UnexpectedCalculationException(msg);
                }
                values[values.Length - 1] = mo(values.Select(item => double.Parse(item))).ToString();
            }

            if (!Operators.TryGetValue(opCode, out mo))
            {
                var msg = string.Format("Cannot resolve calculation operator of {0}.", opCode);
                throw new UnexpectedCalculationException(msg);
            }
            var valueArray = values.Select(item => double.Parse(item));
            var result = mo(valueArray);
            return result.ToString();
        }
    }
}

But what is more interesting is at the supporting class below: -

    internal static class Math
    {
        private delegate T TryExecuteHandler<out T>();

        internal delegate double MathOperatorHandler(IEnumerable<double> values);

        private static double Handle(TryExecuteHandler<double> handler)
        {
            try
            {
                return handler();
            }
            catch(FormatException e)
            {
                throw new IncompatibleDataTypeFormulaException("Unexpected data format.", e);
            }
        }

        internal static double Add(IEnumerable<double> values)
        {
            return Handle(() => values.Sum());
        }

        internal static double Subtract(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  var result = valueArray[0];
                                  for (var i = 1; i < valueArray.Length; i++)
                                  {
                                      result -= valueArray[i];
                                  }
                                  return result;
                              });
        }

        internal static double Multiply(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  var result = valueArray[0];
                                  for (var i = 1; i < valueArray.Length; i++)
                                  {
                                      result *= valueArray[i];
                                  }
                                  return result;
                              });
        }

        internal static double Divide(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  var result = valueArray[0];
                                  for (var i = 1; i < valueArray.Length; i++)
                                  {
                                      result /= valueArray[i];
                                  }
                                  return result;
                              });
        }

        internal static double SquareRoot(IEnumerable<double> values)
        {
            return Handle(() =>
                              {

                                  var valueArray = values.ToArray();
                                  if (valueArray.Length > 1)
                                  {
                                      var msg = string.Format("Expect valueArray of length 1 but was {0}.", valueArray.Length);
                                      throw new UnexpectedCalculationException(msg);
                                  }

                                  return System.Math.Sqrt(valueArray[0]);
                              });
        }

        internal static double Pow(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  var result = valueArray[0];
                                  for (var i = 1; i < valueArray.Length; i++)
                                  {
                                      result = System.Math.Pow(result, valueArray[i]);
                                  }
                                  return result;
                              });
        }

        internal static double Reciproc(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  if (valueArray.Length > 1)
                                  {
                                      var msg = string.Format("Expect valueArray of length 1 but was {0}.", valueArray.Length);
                                      throw new UnexpectedCalculationException(msg);
                                  }

                                  return 1/valueArray[0];
                              });
        }

        internal static double Percent(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  if (valueArray.Length != 2)
                                  {
                                      var msg = string.Format("Expect valueArray of length 2 but was {0}.", valueArray.Length);
                                      throw new UnexpectedCalculationException(msg);
                                  }
                                  return valueArray[0]*valueArray[1]/100;
                              });
        }
    }

In the unit tests, we expected two different kinds of exceptions as such: -

    [Serializable]
    public class IncompatibleDataTypeFormulaException : Exception
    {
        public IncompatibleDataTypeFormulaException(string message, Exception innerException)
            :base(message, innerException)
        {
        }
    }

    /// <summary>
    /// <see cref="UnexpectedCalculationException"/> - Exception indicates that there were some unexpected internal calculation error.
    /// </summary>
    [Serializable]
    public class UnexpectedCalculationException:ApplicationException
    {
        public UnexpectedCalculationException(string message)
            :base(message)
        {
        }
    }

Now that I don't want to make the case complicated by implementing a FormulaParser and FormulaToken all that sorts just for a simple calculator, I opted for the static class implementation named Math. This supporting class is never in my mind that will be shared. It is my internal calculation logic that is for abstraction. It has very high chances that I want improve it in the future.

This class should never be unit tested.

If you do, you will quickly find you do owe yourself for a blame. Unit testing your abstraction logic will pressurize on your responsibility of maintaining highest potential to change API, process workflow, class design and etc. You ended do a lot of work and then undone many work that you were spending day and night maintaining it.

Best practice of TDD said, let those highly changeable code (abstraction logic) automatically covered by just unit testing your public API. If it was not covered automatically, you have written unplanned code and that is the risk for defects or bugs. You will be helped by the coverage to uncover the statistic for this area of risk to you.

Note: The only place that is not well-tested is at UI.

Next, look at the abstraction logic again, there is a pattern of repeatedly throwing exception is needed for most of the function like Add, Divide, Multiply etc, this just comes naturally that I will need some aspect oriented programming for exception handling in this function. Hey, now I need to use AOP design patterns with Code Injection facility.

This is awesomely beautiful isn't it, doing it TDD?

Test Driven Approach - Positive Impact for Developers - Workshop III (Part B)

Now, lets us try to compile and run the test again. As expected, it won't compile at all. We need refactoring to the unit tests at Calculator.Tests as such: -

using Calculator.Core;
using Calculator.Presenter;
using NUnit.Framework;

namespace Calculator.Tests
{
    public class CalculatorMockView : ICalculatorView
    {
        public string Result { get; set; }
        public string Formula { get; set; }
    }
    
    [TestFixture]
    public class CalculatorTester
    {
        private Presenter.Calculator _calculator;
        private ICalculatorView _view;

        [SetUp]
        public void SetUp()
        {
            _view = new CalculatorMockView();
            _calculator = new Presenter.Calculator(_view);
        }

        [TestCase("6", "6")]
        [TestCase("5 + 6", "11")]
        [TestCase("3 - 2", "1" )]
        [TestCase("5 * 6", "30")]
        [TestCase("12 / 6", "2")]
        [TestCase("5 + 6%", "5.3")]
        [TestCase("reciproc(2)", "0.5")]
        [TestCase("pow(2,3)", "8")]
        [TestCase("sqrt(4)", "2")]
        [TestCase("12 / 6 / 8", "0.25")]
        [TestCase("reciproc(2,3,2)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("sqrt(4, 21)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("A + B + C", "", ExpectedException = typeof(IncompatibleDataTypeFormulaException))]
        [TestCase("HelloWorld(2,3,2)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("2 3 2", "", ExpectedException = typeof(UnexpectedCalculationException))]
        public void EvaluationTest(string formula, string expected)
        {
            _view.Formula = formula;
            _calculator.Evaluate();
            Assert.AreEqual(_view.Result, expected);
        }
    }
}

And Calculator.Core.Tests as such: -

using NUnit.Framework;

namespace Calculator.Core.Tests
{
    [TestFixture]
    public class CalculatorTests
    {
        readonly Calculator _calculator = new Calculator();

        [TestCase("6", "6")]
        [TestCase("5 + 6", "11")]
        [TestCase("3 - 2", "1")]
        [TestCase("5 * 6", "30")]
        [TestCase("12 / 6", "2")]
        [TestCase("5 + 6%", "5.3")]
        [TestCase("reciproc(2)", "0.5")]
        [TestCase("pow(2,3)", "8")]
        [TestCase("sqrt(4)", "2")]
        [TestCase("12 / 6 / 8", "0.25")]
        [TestCase("reciproc(2,3,2)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("sqrt(4, 21)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("A + B + C", "", ExpectedException = typeof(IncompatibleDataTypeFormulaException))]
        [TestCase("HelloWorld(2,3,2)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("2 3 2", "", ExpectedException = typeof(UnexpectedCalculationException))]
        public void EvaluationTest(string formula, string expected)
        {
            Assert.AreEqual(expected, _calculator.Calculate(formula));
        }
    }
}

Above, I have also added a few additional test cases where necessary. Now lets us compile and test run them. As expected, they are still failed.

Before we proceed, to correct all these tests, let us revisit what have we done so far?
1. We created solution based on some (small) fundamental architecture design.
2. We created a wire frame demonstrated how this look like conceptually.
3. We then created some unit tests focuses on "how we think the public API would be useful and clean". (API Usability Concern from consumer perspective.)
4. We introduce new projects (core) to ensure we provide better separation of business core logic. (Separation of Concern and Single Responsibility)
5. We then decide again how is the public API of the new project just added would look like. (API Usability again)
6. We implemented the presenter logic, wire up the UI wire frame, and the Core logic classes. (some UI design patterns were introduced for unit tests and better code separation).
7. We refactor the some API within its implementation and its corresponding unit testings in Calculator.Tests and Calculator.Core.Tests. (Refactoring based on needs).

Now, all the above just came naturally as we do our development in TDD fashion.

So, the next workshop, we will be aiming to clear all these unit tests.

Test Driven Approach - Positive Impact for Developers - Workshop III (Part A)

Quickly, we are now at 3rd session. Before we start, I would like to cover a bit on the missing architecture detail. This small application will be (hopefully) tailored into 3 layers design. UI, Core Logic, Data Access (if any). It is not that the requirements asking for so, it is simply some thought to have clear separation such that each component can be more tidy up with better cut of responsibility. Also, I hope I will have an opportunity in the future to bring this example into some further discussion on Scaling Out and Services Oriented fashion.

We have not create any data access layer assembly, but because we created a new assembly, "Core", hence a new Unit Test project for Core project is needed.

Enough said, now lets focus on the assignment. In reality, the calculator will be very unlikely to perform the calculation of such formula at once, "5 + 5% * 50 / 2 + pow(2,3)". Soon as you type in "5 + 5%", the calculator has already responded with the result. (Please verify with Windows Calculator). So, we will probably not cover this within this series of workshop.

Question: Where do we start coding to pass the test?
Answer: Let's start with the UI.

Question: How do we test the code in the UI?
Answer: We need a presenter or controller logic.

Calculator Presenter class develop as such: -

using System;
using System.Text;

namespace Calculator.Presenter
{
    public class Calculator
    {
        private readonly ICalculatorView _view;
        private readonly Core.Calculator _calculatorCore;

        public Calculator(ICalculatorView view)
        {
            _view = view;
            _calculatorCore = new Core.Calculator();
        }

        public void Evaluate()
        {
            _view.Result = _calculatorCore.Calculate(_view.Formula);
        }
    }

    public interface ICalculatorView
    {
        string Result { get; set; }
        string Formula { get; set; }
    }
}

This will need an additional Calculator class within the Core assembly as such: -

    public class Calculator
    {
        public string Calculate(string formula)
        {
            throw new NotImplementedException();
        }
    }

Now we sort of have everything (API) within this application that needs public access modifier done. We need to ensure the new public API sits under the Core project is unit tested as well. In this case, it is very similar to the earlier unit test created in Workshop II.

Next, we focus on how to wire up the UI into the presenter logic. This is what generated by VS 2010.

namespace Calculator
{
    static class Program
    {
        /// <summary>
        /// The main entry point for the application.
        /// </summary>
        [STAThread]
        static void Main()
        {
            Application.EnableVisualStyles();
            Application.SetCompatibleTextRenderingDefault(false);

            Application.Run(new CalculatorForm());
        }
    }
}

And here, we improve the sample WinForm class created during prototyping to utilize the presenter coded, as such: -

using System.Windows.Forms;
using Calculator.Presenter;

namespace Calculator
{
    public partial class CalculatorForm : Form, ICalculatorView
    {
        private Presenter.Calculator _calculator;

        public CalculatorForm()
        {
            InitializeComponent();
            _calculator = new Presenter.Calculator(this);
        }

        public string Result
        {
            get { return textBoxDisplay.Text; }
            set { textBoxDisplay.Text = value; }
        }

        public string Formula
        {
            get { return textBoxDisplay.Text; }
            set
            {
                textBoxDisplay.Text = value;
                textBoxDisplay.Refresh();
            }
        }
    }
}

Till this stage, we already have wire up most of the API needed within this application.

Test Driven Approach - Positive Impact for Developers - Workshop II

First of all, there were couple of useful tools for Test Driven within Visual Studio .NET 2008 or 2010.

These are the tools the I usually use: -
1. NUnit
2. TestDriven.NET
3. JetBrains ReSharper

Note: JetBrains resharper is not free though.

So, after all, how should we start the development based on the requirements?

I usually create a new Test Project first, named Calculator.Tests.

For any TDD development, we should always attempt to write the tests from highest level possible with the simplest code. To achieve this, we should always attempt with the simplest API that we are going to deal with.

So, I would imagine the calculator class to work like this: -

Calculator cal = new Calculator();
var result = cal.Evaluate(formula);

Hence, I created a class within the Calculator.Tests as such: -

using NUnit.Framework;

namespace Calculator.Tests
{
    [TestFixture]
    public class CalculatorTester
    {
        private Calculator _calculator;

        [SetUp]
        public void SetUp()
        {
            _calculator = new Calculator(); 
        }

        [TestCase("5 + 6", "11")]
        [TestCase("3 - 2", "1")]
        [TestCase("5 * 6", "30")]
        [TestCase("12 / 6", "2")]
        [TestCase("2 * 3 - 4 / 5 + 6", "11.2")]
        [TestCase("5 + 6%", "5.3")]
        [TestCase("reciproc(2)", "0.5")]
        [TestCase("pow(2,3)", "8")]
        [TestCase(new [] {"A + B + C", string.Empty},
                    "Incompatible data type evaluation test.",
                    typeof(IncompatibleDataTypeFormulaException))]
        public void EvaluationTest(string formula, string expected)
        {
            var result = _calculator.Evaluate(formula);
            Assert.AreEqual(result.ToString(), expected);
        }
    }
}

Now with have almost demonstrated most of the features needed using various test cases based on TestCaseAttribute of NUnit 2.5.

You may be asking, so, this is the business logic, where is the UI testing? Isn't it we must start from the UI?

Yes, you got it half spots on. We must start from the UI (highest level), however are you sure this calculator class is just a business logic class? We shall see then.

Now, lets try to fix all the compilation error by creating some fundamental skeleton for it to compile.

public class Calculator
{
    public double Evaluate(string formula)
    {
        throw new NotImplementedException();
    }
}

public class IncompatibleDataTypeFormulaException:Exception
{
}

Now that we have written the unit tests and the compilation is done. We should test run it and see all of them failed.



As expected, all test cases failed. I would be extremely surprised if any of this passed. :)

So, the next assignment is to get them green.

Test Driven Approach - Positive Impact for Developers - Workshop I

Start from this post and over the next few posts, I will be covering the Test Driven Approach - Positive Impacts for Developers.

Impacts: -
1. Think like a user
2. Clean APIs Design
3. Improves Abstractions
4. Better Used of Patterns
5. Improves Separation of Concerns.

Let starts with some fundamental user requirements on a calculator.

I can do fundamental calculation, +, -, *, /, sqrt, %, pow and 1/x.
I can use whole numbers or rational numbers as the input.
I can defined the result format within the format of whole number or rational number.
I can specify the number of decimal places for the results.
I can define a formula and save it.
I can load a formula and calculate using the loaded formula.
I can only specify the values for variables within the formula.

There is all we have for the requirements.

So, if we're to start right away, we should have start writing the unit tests first.

A quick glance into the requirements, it seems quite straight forward to accomplish.

Questions: -
1. How many of you already start thinking of classes for logic separation, like Adder, Subtractor, Divisor and Multiplicator etc.?

2. How many of you already start writing unit tests right away based on the requirement defined?

"If there is anyone falling into those traps, you're wrong!" The requirement wasn't quite defined into a stage where this can be started.

Based on the requirements given, we should have at least asking how would it be deployed? Web Services, Windows Services, Web Apps or Windows Forms Apps for instance.

So, to make it more simple, this is a Win Form application, just like the calculator in Windows.

New User Requirements: -
"I can use it like many windows desktop application."

So now, we are good to start, however, not unit testing yet. It is the UI wireframe (prototyping for GUI, i.e. Forms.) Here it is: -



Now, we can start writing some tests, that will be in workshop II.

ETL Sorting or SQL Sorting (ORDER BY)

ETL Magic

---

Often hear about ETL (Extract, Transform, Load) solution for better data integration between applications, systems or solution within an enterprise. What are the major consideration before choosing an appropriate ETL solution?

Sorting, Merging, Unification, Transformation and Loading are quite commonly happen within Data Integration stage. Often, we heard of NSort, CoSort (IRI), SyncSort and others famous 3rd party ETL solution and tools. Each of these offers some great performance if not exceptional in their respective technical strength with many data integration efficiency (Terabytes of data size) bar being set higher and higher.

EnTerprise solution missing Link?

---

Quite honestly, many solution architects would think these ready made ETL solution is best used to glue their data movement processes within the sub-solutions altogether to not just making data extraction, transformation and loading much faster (conceptually), but also act as a missing link to gel data from many independent components within the enterprise solution to ensure they get the right data in timely fashion before going live.

Question of the day

---

What would be the topic of discussion today is rather, will those 3rd parties solution offers best performance value to the enterprise solution and often considered as the missing link in the solutions integration?

ETL Rules of thumb: -

---

If data is already in the database and you would want to extract all these data into another database, consider doing the sorting and filtering related tasks within your database server itself. Ordering, filtering and unification tasks within the database server is always consider as "as good" when compared to the other ETL solution.

If the data has to be somewhat inserted into tables within database, consider doing sorting, unification, merging (joins), filtering, data enrichment (meta tagging) and etc. altogether within the database server. Again, the same principles applied here. Database server can handle all those as good as the third parties ETL.

If the batch data integration process can be avoided by putting it into an incremental data integration approach, by all means doing it in an incremental fashion with proper application/data partitioning.

If the data is migrating from one database platform into another, try to look for some out of the box integration solutions. For example, SSIS for Microsoft SQL Server. (SSIS is free and has also recently breaking some integration records.)

Never ever thinking of re-inventing your own ETL solution, from ground up, if you are not intending to step into the red ocean, ETL business.

Consider 3rd parties ETL solutions when the data source is originated from a non-database model and the data target is another non-database model.

Consider 3rd parties ETL solutions when majority of the data transformation, extraction, unification, merging, filtering and sorting involves many different types of data sources.

Consider 3rd parties ETL solutions when it has more than one intended data target that are of different platforms.

Take Away

---

SQL ordering, filtering, conditional relationship/merging (joins) are almost equally as fast as the 3rd parties ETL tools.

No real magics with ready made ETL solutions, it is the solution integration architecture that counts.

Fancy solutions will cost more in terms of investment capital, learning curve, maintenance efforts, business opportunity, end product selling prices (to your customers).

Side Effects of Linq Select Statements due to Deferred Execution

I have a function to read data from a delimited text that is splitted into a string array. To the caller didn’t specifies the columns to extract, it extract all columns, and only extracts the specified one otherwise.

Assumed that I have a delimited text that contains 2 columns only: -

public DataField[] Read()
{
    // splits the text according to the defined delimiters.
    var data = _reader.Read();
    // when no columns specified, take all column from data.
    if (_columns == null)
    {
        var counter = 0;
        // save as the instance fields to avoid recompilation.
        _columns = data.Select(item => new ColumnInfo
                                        {
                                            Index = counter++
                                        });
    }

    return data == null
    ? null
    : _columns.Select(item => new DataField
                                {
                                    Column = item,
                                    Data = data[item.Index]
                                }).ToArray();
}

To my surprise that this Read method keeps throwing ArrayOutOfBoundException after the first attempt. Can anyone guess what’s wrong? :) (Answer: scroll down please)

After the first call, the enumerator is wellover (at its end element). However, IEnumerable created (which I thought it was) from a Linq statement is not a real creation of IEnumerable instance.

_columns = data.Select(item => new ColumnInfo
                                {
                                    Index = counter++
                                });

This statement is rather associating _columns field to the Linq statement (like a callback, I think). The subsequent Read() call would continue to invoke the lambda expression within the linq statement for next element, which created column with Index=3. (This also explains why linq’s performance is good without much overhead).

To workaround this side effets, one need to really tell the Linq function to create a real array or list for caching and the side effect will be gone.
Solution as such: -

public DataField[] Read()
{
    var data = _reader.Read();

    if (_columns == null)
    {
        var counter = 0;
        // ToArray() will tear off the callback by caching a real array, not callback.
        _columns = data.Select(item => new ColumnInfo
                                        {
                                            Index = counter++
                                        }).ToArray();
    }

    return data == null
        ? null
        : _columns.Select(item => new DataField
        {
            Column = item,
            Data = data[item.Index]
        }).ToArray();
}

Read Jon Skeet article on Human Linq

Also here about side effects with select

TDD: Only Unit Testing Public

Many times when I spoke to people about TDD and Unit Testing, quite surprisingly I will be trapped into a debate of "Unit Test Public only?".

My quick answer is "Yes".

Reason?
Internal, Protected, Private "properties" of an Assembly or Class are the abstraction for each of them respectively. Unit tests should never cross the border to break the abstraction intend of a design.

DISCLAIMER: "properties" I used here refers to methods, subtypes, property (fields) etc.

Come to think of this topic again, why would one asked the question of such like? (unit test public?)
My only view is that, these may be the people who are writing real implementation first before writing the tests. They knew the design and implementation before even writing unit test and they knew the importance of the internal implementation and they would like to secure it (secure logic implementation intend). Are these suggesting TDD practice? (Refer to my earlier TDD post HERE).

If one started to think of the application (usage) of code to be developed first before thinking how to develop it (or design it), one should start with the accessible content. (Which is public content). Get the usage right is the prime criteria for TDD and also many software quality assurance measure. From the righteous usage of the public "properties" corresponded with right expectation and that makes the "right" deliverable.

So, spend more time thinking of what may be "right" for the time of writing, and ensure unit tested the "right" intend that you would want to enforced. Internal implementation are abstraction that meant for future re-development or refactoring or enhancement that usually and very likely to change from time to time without prior noticed.

This way, you have done your team and yourself a favor by created some maintainable unit tests.

MSSQL Common Table Expression (CTE)

What is Common Table Expression?

• Can be thought as a temporally result set that is defined within the SQL execution scope.
• Usage: -

1. Create Recursive Query
2. Substitute for a view
3. Enable Grouping by column that is derived from a scalar sub select.
4. Referencing the resulting table multiple times.
5. Available since SQL Server 2005

Syntax: -

WITH [(coloumn name [, column name...n])]
AS
(
/* anchor member definition */
UNION ALL | UNION | EXCEPT | INTERCECT
/* recursion member definition*/
)



Statement to use with CTE include INSERT, UPDATE, DELETE, CREATE VIEW.

CTE can be used without recursion member and that will serve very well as the temporally result set for further joining.

Example: -

/*************** BEGIN OF QUERY **********************/
USE NRule
GO

WITH DocumentReport
AS
(
SELECT a.id, a.[Description], a.Name AS [DocumentName], c.Name AS [AttributeName], d.value
FROM Documents AS a
INNER JOIN DocumentMetaAttributePathIndex AS b
ON a.id = b.documentId
INNER JOIN MetaAttributes AS c
ON b.attributeId = c.id
INNER JOIN AttributeValues AS d
ON b.valueId = d.id
)
SELECT *
FROM DocumentReport ORDER BY id
GO
/*************** END OF QUERY **********************/

CTE would be best use to query of outcome that is expanding dynamically. Typical example is to trace the tree or graph data structure represented under adjancent list table design pattern within relational model of a database.

Example: -

/*************** BEGIN OF QUERY **********************/
USE NRule
GO

DECLARE @documentId bigint
DECLARE @ancestorId bigint
SET @documentId = 1
SET @ancestorId = NULL;
WITH AttributeHierarchyReport(AncestorId, AttributeId, DocumentId, ValueId, LEVEL)
AS
(
-- Anchor member definition
SELECT meta.AncestorId, meta.attributeId, meta.[DocumentId], meta.[ValueId], 0 AS LEVEL
FROM dbo.DocumentMetaAttributePathIndex AS meta
WHERE (@ancestorId is NULL or meta.AncestorId = @ancestorId) AND
(@documentId is NULL or meta.DocumentId = @documentId)
UNION ALL
-- Recursive member definition
SELECT meta.ancestorId, meta.attributeId, meta.[documentId], meta.[valueId], LEVEL + 1
FROM dbo.DocumentMetaAttributePathIndex AS meta
INNER JOIN AttributeHierarchyReport rpt ON meta.ancestorId = rpt.AttributeId
WHERE (@ancestorId is NULL or meta.AncestorId = @ancestorId) AND
(@documentId is NULL or meta.DocumentId = @documentId)
)
SELECT AncestorId, AttributeId, DocumentId, ValueId, Level
FROM AttributeHierarchyReport ORDER BY Level;
GO
/*************** END OF QUERY **********************/

However, watch out for the data redundancy that is appearing in the query result set. Redundant query result set within the recursive CTE proven to be a killer to your database application using it. It grows the result set sometimes exponentially until it kills the application by running out of memory.

Tweak is needed to CTE when redundant row sets happened and if you care about your application, never live with redundant row sets by either further tuning your CTE or improve your Database/Table design to help better/efficient use with your CTE.

Example: -

/*************** BEGIN OF QUERY **********************/
/* this version is using additional column ParentDescendantId to enforce root node validation at anchor member */
USE NRule
GO

DECLARE @documentId bigint
DECLARE @ancestorId bigint
/* @documentId = NULL would mean to get all attributes for all documents regardlessly */
/* alternatively we can use the @documentId to narrow down to a specific document attribute tree for retrieval */
SET @documentId = 1
/* @ancestorId = NULL would mean to get all attributes regardlessly */
/* alternatively we can use the ancestorId to narrow down the attribute tree for retrieval */
SET @ancestorId = NULL;
WITH AttributeHierarchyReport(AncestorId, DescendantId, DocumentId, ValueId, ParentDescendantId, LEVEL)
AS
(
-- Anchor member definition
SELECT meta.AncestorId, meta.attributeId, meta.[DocumentId], meta.[ValueId], parent.attributeId, 1 AS LEVEL
FROM dbo.DocumentMetaAttributePathIndex AS meta
LEFT JOIN dbo.DocumentMetaAttributePathIndex AS parent ON meta.AncestorId = parent.attributeId
WHERE (@ancestorId is NULL or meta.AncestorId = @ancestorId) AND /* Ensure the ancestor id is matched */
    (@documentId is NULL or meta.DocumentId = @documentId) AND /* Ensure the document id is matched */
    (parent.attributeId IS NULL) /* Ensure there is no ancestor, Root node only */
UNION ALL
-- Recursive member definition
SELECT meta.ancestorId, meta.attributeId, meta.[documentId], meta.[valueId], parent.DescendantId, LEVEL + 1
FROM dbo.DocumentMetaAttributePathIndex AS meta
INNER JOIN AttributeHierarchyReport parent ON meta.ancestorId = parent.DescendantId /* Ensure all is child is recursively joined */
)
SELECT AncestorId, DescendantId, DocumentId, ValueId, ParentDescendantId, Level
FROM AttributeHierarchyReport;
GO
/*************** END OF QUERY **********************/

Don't Limiting Yourself Much Than Your Users or Clients Do.

Many times, before planning on a new software release you often hear the voice of backward compatibility, upgrade costs, additional component dependencies, operating platforms upgrades and etc. from the team. Are these important? Yes they do. Are these debates necessary before the plan of a release? Absolutely not. We need to look at it and aware of it, but lets not debating it now.

A new software release is to ensure it maintains attractiveness and sounds of improve workability, usability, flexibility, integrativity, scalability, security, martketability and many more.

Before one starting to look at the release plan or product backlog, and if you have already be the first to attack on your own team, you are not doing your team a favor, but holding them back to be closer with FAILURE and not able to deliver.

One phrase that I often hear and it is always true is that "Many times, we often limiting ourselves much more than the customers or users do."

Wake up guys! Be wild and creative when innovating of what you want to do to your software during visioning stage, and only come to be more focus and cautious for not over committing when you are planning and delivering it.

Overly conservative does no help of bringing innovations. Bottom line, don't limiting yourself much than your users or customers do would always be a golden phrase to follow.

Test Driven Development

Five key principle questions must be asked all time:

1. What Do I Want to Accomplish?
2. Does It Do What I Want?
3. Does It Do What I Want All of the Time?
4. Can I Depend On It?
5. Does It Document My Intent?

TDD Process Itinerary

1. Obtains Project Specification Documents to identify specification needs, usually the action from the UML CASE diagram. (What needs to be done?)
2. Write the test procedures and test methods (Unit Test) to enforce it achieve the specification needs as how it was documented in Project Specification Document. (Does it do what I want?)
3. Write the code just to pass the test.
4. Ensure and maintain the tests from never fail at all times. (Does it do what I want at all time and Can I depend on it?)
5. Re-factor the code for readability and maintainability, while ensure the tests remain intact to the specification. (Does it document my intent?)

Few ways to achieve unit testing: -

• Dummy
• Fakes
• Mocks
• Stubs

To be continue...

Top 10 Unwanted Characteristic of a Developer

1. Egoness
Example: -
We are senior and we have many years of exposure and we know what we need to do, please keep your explanation to yourself.
Outcome: -
Fail to grow as a team, fail to deliver or deliver with ULTRA high defects. Copying design ideas from others but not bothering to ask and resulting in sub-standard code, but proclaiming that is the beauty of it when challenged.

2. Year-istic Mathematician
Example: -
I want a lead role because of my many years of exposure. I work with many big profound companies and I derserved the lead role.
Outcome: -
Not able to justify the pros and cons and delivering sub standard work or not delivering at all.

3. Favor Quick Wins over Quality
Example: -
Deliver the max within the shortest time is the primary concern of the company to make money. Patterns and design are purism and that often complicates the development.
Outcome: -
Fails to grow as a team or individually. Produced sub standard codes that needs major rework whenever enhancement needs imposed.

4. Never Admit Own Mistakes
Example: -
The bugs had happened because the calling function never do a parameter validation and error checking.
The late delivery was largely due to inconsistent and or incomplete requirements.
Outcome: -
Never learn from mistakes and similar problems reoccuring over and over again. It is quite clear that if it is not your problem but it somewhat impact your work delivery or team goals, it is everyone's problem and deal with it if you can, escalates otherwise.
To be continue....

.NET Application Slow?

Many people complained about .NET application being slow. First question first, do you know how .NET work internally?

Pickup a good book like CLR via C# (Jeffrey Ricther) or Framework Design Guideline (Cwalina, Abrams) would help.

If you don't understand how GC (Garbage Collector) works, please do yourself a favor.

If you find yourself writing lots of codes (and redundant codes), do yourself a favor reading Design Patterns.

If you find yourself adding lots of project/assembly references into a project, do yourself a favor reading books about SoC (Separation of Concern or Single Responsibility or Architectural Consideration).

If you find yourself fixing lots of bugs that subsequently introduce more and more bugs (defects), change your development engineering process but adopting TDD (Test Driven Development) approach. (Unit Testing)

I'm hoping to have touch more indepth details of these topics in the coming post.