Synersov Alive

Micro Management vs Macro Management

2011-11-14T03:02:00.000-08:00

More than often, we heard of these terms. How many of us try understand these definition and see how it impacts and influence the day to day activities.

By Definition

Micro Management - Require one to enforce every of his/her staffs to report on their progress very frequently and reluctantly to delegate decisions. Also known as Bully Approach.

Macro Management - Delegate decisions to his/her staffs and monitor on the progress less aggressively. However, it is often that one delegates too much of decisions without some necessity knowledge of work.

Often we learnt about a leader when we start working in a team. We then come across many different style of management. Each style tells us, indirectly, what type of leader we work with.

Micromanage suitable only if: -

Start up company
Incompetent team members
Mission critical but proprietary knowledge of work

Manager with micromanage style suggests that he/she is still a better person to take on every lower level tasks assigned. Company with managers of tendency to micromanage suffers opportunities of losing their better employee to be a manager who does not manage well but love detail work.

Macromanage suitable only if: -

Well established company
Not only competent but knowledge independent team members
Generally all type of works

Manager with macromanage style suggests that he/she trusts the team members being knowledge independent and capable of producing good work. Decisions and rights are trustfully given with minimal supervision. However, that may also suggest that many manager involves in Macro Management style would lost his initial stint within the business and overly relying on his/her team members which results in decision dependencies and politically driven decision/outcome. Company must love these managers as they made themselves out of job while growing people to out cast themselves.

Conclusion

While micromanaging managers suggest to delegate only risks to his/her staffs and claiming all credits to himself/herself when succeeded, macromanaging managers suggest to delegates credits to his/her staffs and taking the risks on their own.

Either way, you are either too good only to yourself or too good to your people. That only leads you into one way, failure! Yes, you got me right, it is failure! With micromanaging, higher management would have learnt and noticed that they can't scale in your department. You are too important that they need to find someone to manage you. You just missed the promotion by that. With macromanaging however, everyone is doing right and perform well enough to replace you, because you made them too good to put yourself out of job. You have higher pay then others, and why should I pay more if they are equally good as you with better details, but miles cheaper?

You need to struck a balance in your style. You have great people, do more macromanagement and learning the details from your team via some suitably application of micromanage. This way, you won't be thrown away by your people in your business knowledge, while letting them to take their deserved credit.

If you have average people, depends on how much the average is, apply the micromanage as you see fits while not letting the opportunity to slip away for your people to learn and excel with some delegation through macromanagement.

Speaking of a company perspective, I would love to have as many macromanaging managers than micromanaging managers. At least, I'm not tightening up my options on resources.

SCRUM - Manage Your User Story

2011-04-12T05:46:00.000-07:00

Background

We usually have some number of user stories to deliver in each sprint. Often, the user stories were created to our best knowledge about some business values that we are going to deliver. It then gets updated with clearer detail over the time as it turns higher at priority over the time. However it is very often that not all details can be gathered before we started to work on a user story. Very often too, there are some new discoveries as we progressing to complete a user story. These are risks, the risks may be preventing a user story from getting complete. If it prevents the completion of the current user story, how we handle it? Can I have a story extends all the way to the next sprint or couple of sprint?

Manage your new found risks in a user story

In agile, we don't like a large user story that spread well over several sprints without a concrete indication of where we are. This does not manage the risks well. However, if it is a main (big) user story that has several not too large but measurable sub user stories where each of them can be completed either concurrently or independently within a sprint, I believe that is very much acceptable and the way I *think* it should be. This makes the user story, whether the main user story or its sub user stories, more accountable, quantifiable and manageable. It then makes things transparent as to when can I expect certain deliverable based on what we know best now. However, many people would argue that this is in fact a dependency of user story and the dependency management usually needs project planning and that is not agile after all. If you think so, you have mistaken my points I am trying to make here.

Say a developer is going to deliver a user story "A" in this sprint and halfway thru the process, he discovered an impediment, say "A'", that prevents him from getting user story "A" done. He has no additional bandwidth or capacity of getting "A'" done this sprint, so he has discussed with SCRUM Master and it seems nobody has better knowledge than he does to handle it, so what a SCRUM Master would normally do is to get Product Owner into this discussion. If nothing much has changed, what should the Product Owner do to this User Story "A" and the impediment "A'"?

As I would suggest: -

The Product Owner should split the user story "A" into 2 separate sub user stories, say "A1" and "A2", and ensure it is associating back to the original user story "A".
- Eg. A = A1 + A2
Then, ensure in user story "A1", it covers what we need to get done initially are covered with some mocking functionality of "A2" in a very high abstraction (Hard coded may be?)
- Eg. A1 = A1' + A''
- A1' => What we supposed to do initially in A, but with a bit changes in abstraction to cater for the coming impediment of A' we discovered during this iteration.
- A'' => Mocking functionality or resolution to impediment A' that we discovered.

Then within this sprint, we can have the developer to complete the user story A1 and moving the user stories A and A2 to next sprint or re-prioritize as it sees fits.

Questions

Are we not agile?
- No, we are. We still follow "just in time" (as we discovered it) and "just enough" (just the mocking of functionality for now) and "just because" (mocking because we can't handle it now, but later)
Are we doing project planning in advance to identify the dependency?
- No, we don't. We let it happens naturally without factoring in any additional effort in advance to predict.
More importantly, do we need to emphasis on Agile to be Agile?
- No, Agile is the thought process of getting things done and SCRUM is a framework to do things in agile. They are like best practices recommended to be followed, however you must only follow the practices if that makes sense to you. Ultimately, it is the principle of being continuously transparent, lean in thinking, ready for changes and managing risks with adaptive sprint planning that eventually driving us to the release we planned that makes sense to the business.

Hints

We need a real worker in Agile and SCRUM and not a philosopher.

SCRUM - Definition of Done?

2011-03-30T05:16:00.000-07:00

Background

Definition of done in software development companies varies one after another. Some company has additional constraints to fulfil while others may be very little. Definition of done is crucial and one must do it right and customized according to the needs to ensure continuous deliverable. For instance, in TDD environment, we usually need some unit tests per feature introduced with sufficient tests coverage that is predefined (usually > 90%). Next, we need some code review and have the code checked in after the review and yet has the Continuous Integration Build completed and all unit tests passed.

Do we need one in SCRUM?

Certainly we need one. Indeed, we need it to a much wider extends. In SCRUM, we create user stories and then planned out the deliverables in the release plan and the sprint plan. At the end of each sprint, we will mark whichever user stories that are "done" as done. The business users can then expect a somewhat "workable" solution to be delivered.

The question now is, how much do we need to constraint when we are in SCRUM? How much do we want our business users to be able to expect from the delivered "workable" solution at each sprint?

If a solution that works but may be with many defects due to not properly tested is considered acceptable by the business users as a potential deliverable, you can then schedule your QA tests to come in after the sprint!

In many occasions, in fact in SCRUM, this is not acceptable. You cannot say you've completed a user story where it is highly potential with defects, especially the functionality or workability defects. When you mentioned a done to the user story, we are expecting it to be stable, workable with minimal functional defects and met some level of quality measure.

So in Agile or SCRUM, we do tests much sooner and users stories that failed the tests can never be made DONE. We bring the test team into our SCRUM team and they are in our team. Soon as we completed the development work, the tester will do the tests according to the user story's needs. We don't delay the tests and wait till the end-of-sprint and packaged the deliverable to test. Many SCRUM teams failed, one of the key reason is that we do post sprint test approach. You always need to factor in additional period of time for the tester to tests and the defects of this sprint can only get fixed next at sprint. Your SCRUM fails, you are running waterfall approach (miniature waterfall style). So, for the SCRUM Master, please change this to in-sprint testing for the better success; for the Product Owner, please voice your concern of having it (defects and quality) delayed to next sprint. You don't always have that next sprint and worst, you are always wasting a sprint in addition!

Compulsory Items in SCRUM DoD List

Automated Unit Tested with Good Coverage (above 90%)
Continuously Integrated and Successful Build
Automated Integration Tested
QA Tested (with Test Team)
All Defects Fixed.

SCRUM - Endlessly Growing Product Backlog

2011-03-29T05:35:00.001-07:00

Background

Often when we have run our SCRUM over a period of time (a year or two), we noticed the product backlog gets expanded at an increasing velocity. Up from some 1 or 2 new user stories introduced to several tens of new user stories introduced within a sprint. Is this a good indication or worries that one should look into?

User stories within product backlog indicates some missing business features or value that one must look into and plan out properly to ensure the business satisfaction. This usually has to be inline with the product roadmap and how we want to grow the product over the time. This roadmap can be changed as time comes, to ensure it meets the current and near future (not too far in the future) business needs. Remember that phrase we talked about, Just in time", "Just enough" and "Just because"!!!

Why It Happens

Everyone raises a user story they "think" it is value added.
No clear inidcation of product roadmap telling where we want to be within a fixed schedule.
No planned out of fixed schedules.
Overloaded with too many technical driven user stories

How to Turn It Around

Who can raise user stories?
Where we want to be at a bird's eye view?

Just in time", "Just enough" and "Just because"!!!

Where we want to be at a bird's eye view with a minimal lookahead?
Should we raise technical user stories?

Never write you own Message Queuing Framework

2011-03-18T00:48:00.000-07:00

Software development is fun. Especially true if you are developing an Enterprise Application. There will be many challenges, world class challenges! High throughput, efficiency and low roll out cost and many sophisticated and complex business logic and workflow that goes beyond what you have learned and practiced in the past. If you are not ready for them, you better look elsewhere.

Very common in enterprise solutions to have distributed application support that require high response time and yet reliable communication and services to deliver the requests. Secured, durable and reliable messages encapsulating business operational messages that get processed at the distributed end, usually the server tiers, are common. These are normally done with many ready made Messaging Framework, such as MSMQ, AppFrabic Service Bus, Enterprise Service Bus, WebSphere MQ. There are also a couple opensource messaging framework like RabbitMQ, ZeroMQ, AMQP.

We have more than enough varieties in MQ selection however, there always some techy developers or architects would like to invent one them-self within their own team. Often, the discussion is surrounding the fact of freedom and lightweight modules that suits best into the solutions, teams or the company. It is always good to have the experience of developing such framework in your resume, but that is often bad for the team, and company.

Never invent one there is a ready made solution.
It takes several years for a framework to be matured and tested against many genuine requirements within the businesses and made many fixes, enhancements, simplification, optimization, customization, usability and security tests while accomplishing the needs of being orderable, durable, reliable and securable. Developing one yourself is a short sighted move. You are ignoring the facts that a framework takes millions of hour and efforts to be matured and testified. You are thinking that one man view is always better than the effort of a group of people. You might be right for the first few months, but as your solution becoming more matured and many more challenges coming onboard, you will quickly find that you were shooting at your own feet for the decision you made.

You often find that your invented MQ framework does not scale as good. Has many shortcomings in throttling, failed-safe, and security. Does not have a dead letter policy supports. Loosing the sight of dynamic expansion of queues. Priority overwrites and poison message control. Worst when you have no idea to maintain/support disastrous recovery. Worst when you find your own MQ framework does not support distributed transaction, transaction flow (distributed but integrated transaction model) and does not log properly and not autonomous. Your solution may ended up just a distributed but serialized messaging gateway that processes messages from different sources in a single queued fashion and starts to suffers when a business starts growing large.

Other than business factors consideration, you owe your team or your company at large by requiring many innocent teammates to endlessly support and fixing defects that were common but long fixed in the matured framework. You start to find yourself heading to where other frameworks is doing by doing more work and getting less throughput. You burn many pricey man-hours for the company to get less. You tricked your team members with more long working hour to support and defect fixing to deliver sub-standard solution to the users. You make your product less competitive to the competitors and you shifted the team focus from product development to platform components that are no better but long available. You sincerely given your competitors a chance to close up your leading role or gap in the industry. You put your team to loose enthusiasm. You bid your company and the team to the losing end to only build a good resume in you.

From the business, team, and product perspective, there are no reason for one to develop your own messaging framework. Unless you would want to compete with the giant in the market, this is not your cup of tea. Just stay away from losing sight, you should and must maintain your business focus and continuing your industry leading role.

Final words, don't reinvent the wheel if this is not your strength and does not record in your business roadmap. Do what you do best and leave the worries to the experts.

Test Like A User

2011-03-14T21:43:00.000-07:00

Often, for a developer to be good, one must think like a user. Same goes to the testers.

So, what it says, before you start writing your test cases, you must think, "If I am one of the users, I would want something to work like this. It must not allow me to proceed if I ever mistakenly committed certain things. Best if it can give me some alert (warning).". I know many would say to me, this is a user requirement my friend, and testers are testing the product based on the requirement! Well, yes and no.

Do you how many good requirement surfaced during test phases. Users may have overlooked, and testers is to tests whether that is going to be painless experience for a user other than the workability. I know many users created many requirements which eventually they themselves find it not useful and painful to use. They may have mistaken one or two steps which unfortunately were not noticed.

As a tester, you must ensure you know the context of the topic you are testing. If it involves Integration Layer with Integrations testing, you must ensure you understand how this integration is done. Why it was done like that (the flows and steps) and can it be better?

Many times, the tester came back to the developers, and asked things like: -
"Why when I'm migrating my Database from the console application, I need to tell which server I'm migrating to and what new name my database is going to be?"

"I'm testing on a command to restore a database which optionally takes a database synonyms fixes. Why must I specified the synonyms when I'm restoring it to a different database name?"

I often ask my testers right after they asked questions. Such as, do you know what is a database migration? Do you know what is a synonyms?

Sadly, the answers I often get from the testers are "No" or something like, "I'm not certain" or telling me something that is totally irrelevant. I'm just thinking, how can you do the tests as a tester if you don't even know what you are going to expect and how it is going to behave. Worst when one can't even tell me what errors can they expect as a tester.

Testers are not a dump "machine" who runs routine works and verifies the outcome with a predefined results only. It is so much broader so the testers can act like a user and telling the developers/users/BA or even the genuine users themselves with something like this: -
"Look, what is done is good, but I certainly think as a user I would want something additional to make it more useful (usability) and informational (support). I understand this is not as important compared to many other features, but certainly we need to be more painless to use it."

Hey testers, wake up. Stop your routine boring tasks of matching results. Try step out of the box and thinking like a genuine user and use the product like you are owning one later. You can make a different!

SCRUM Principles

2010-12-02T20:15:00.000-08:00

Background

Attended the SCRUM Workshop, in short, it is a must for all developers/Architects/PM/Program Manager/Tester/BA.

SCRUM Values

In SCRUM, we value: -

Individuals and interactions over processes and tools
Completed functionality over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

How SCRUM Helps?

Works Transparency - Everything is transparent to the corresponding stake holder, customers and even to the team. Burn Down Chart, Team Velocity.
Self Empowered and Organized - SCRUM Team work as a whole to estimate and agree on deliverable achievable.
Collaboration - The transparency of the progress, status and facts allows the SCRUM team to collaborate and interact more to the stake holders, customers on change plan without over optimistically committing to the request too easily.
Recognize the needs to change - The only thing remain unchanged in this world is the change. It recognizes it and put focus on what is best known currently and do it while anticipate future changes is needed.
Incremental Release - Encourage frequent release with each release being workable deliverable as a whole, in a smaller but manageable incremental scope.
More and frequent planning - SCRUM does more planning, Product Road Map, Release Plan, Sprint Plan. However, only the highest priority which needs immediate work gets the detail plan, tasks, estimation and etc.
More Meeting - Daily SCRUM meeting. Daily SCRUM meeting on What is done, what is not, what is on your way.
More Frequent Reviews - Sprint Reviews at the end of each Sprint on what is completed and what is not by adhering to Definition of Done.
Sprint Retrospective Meeting - After the sprint reviews with Product Owner excluded. Discussed on what's good and bad as project is still ongoing. To know the problem earlier, learn and improve as a continuous effort and avoid repeating the same mistake within the coming Sprint.
Product Owner - Single person to have final authority representing the customer's interest in backlog prioritization and requirements questions. This ensure no missing link or tear down at communication.
SCRUM Master - As a facilitator to work with the product owner and the team to orchestrate the SCRUM ceremony.

Untrue SCRUM/Agile Myth

SCRUM does not do documents or reporting for communication!
SCRUM is only focus on the worker (developers) not the business!
SCRUM does not need a Leader!
SCRUM makes impossible turns possible!
SCRUM prevents problems and it is a "silver bullet"!
SCRUM is self organized and self empowered, so, "Do whatever you like!"
Agile Fragile! No planning and architecture needed.
Agile is just an iterative and incremental process.
Adopting Agile means betraying PMI.
Agile means no/avoid commitment.

Reality to Untrue SCRUM/Agile Myth

Reality to Myth #1:
In SCRUM, we emphasize on automated and integrated workspace. We produce artifacts such as: -

Product/Release/Sprint Back Log
Product/Release/Sprint Burn Down Chart
Team Velocity Matrix
Estimation

These artifacts are meant to be showing to everyone, where we are, how are we doing and how far we are from the sprint/release/product goal. Product backlog with each itemized entry prioritized showing the road-map of a given product and the planning of getting it done accordingly.

Reality to Myth #2: -
SCRUM is highly business value driven approach. It focus on business values to the customer than the scope or what customer "think" they want. It ensure the customer is not buying just a set of feature scopes, but the business values to the customer, sooner and earlier. Often, we see projects delivered all the scopes their respective customers wanted but the customer isn't happy in the end because it does not fulfill the business need. This is what SCRUM is trying to avoid.

Reality to Myth #3: -
In SCRUM, we are changing the game plan. Managers are too used to Directed management style. These managers need followers. They ensure the best arrangement of work to candidate who they think best working with. Often, their best people usually does not work best under such "arrangement". Think again, the best people who does well in their work are already put on to manage role, so is there anymore best people? So, in SCRUM, we coach, mentor and lead by example. We value facilitator management than directed management. This requires a capable leader with great leadership skills and interpersonal skills to make possible.

Reality to Myth #4: -
In SCRUM, we value transparency and we focus on start delivering sooner with multiple release that ships incremental of workable solution subsets. This usually help to discover and manage the impossible sooner and allow the customers to justify their decision quicker and change of game plan earlier. Hence it helps to see the fails sooner to avoid greater lost because the nature of identifying risks/impossibles sooner, the customers often can then make justifiable decision to counter the issues earlier and has higher success rate than those that can't help in identifying risk earlier.

Reality to Myth #5: -
In SCRUM, no way you can prevent problems. However, its emphasize of being transparency will let the problems to surface earlier than one would expect. These surfaced problem are Impediments in SCRUM which requires the SCRUM master with great skills to help facilitating with the team. This impediment may be impacting the productivity of the team but SCRUM allows it to be surfaced sooner and dealt with earlier and quicker. However, to some extent, people often associate this nature as a SCRUM problem itself.

Reality to Myth #6: -
In SCRUM, we need a highly discipline members that stay focus and continue productivity with consistent pace for predictable estimation. SCRUM team works closely achieving Goals and Expectation put up front by Product Owner who represents the Customers with Customers interests and expectations. SCRUM Master as a facilitator is trying to collaborate between Product Owner and the team and orchestrating the each SCRUM ceremony. Everyone has their role and game plan and this discipline must be closely adhered to.

Reality to Myth #7: -
In SCRUM, we do more planning. Product/Release/Sprint Planning. We do more risk management. Product/Release/Sprint Burn Down. We do more reviews and we do more retrospective meeting (postmortem) than anyone else, however this are done interactively within each sprint and making us so efficient in dealing with problems because we see problems sooner, efficient in learning because we see our weakness earlier, and etc. In SCRUM we need architecture to work best, however, we avoid architecture details being too detail too soon because we recognize the only thing remain unchanged is "change".

Reality to Myth #8: -
In SCRUM, we measure our productivity velocity based on facts (the earlier Sprints' velocity from the burn down) and commit to tasks within our capability. We aimed to deliver potentially shippable solution with a current subsets of product features that delivered business values where our customers need it NOW, not just want. It is not as simple as incremental, but it has a lot to do with customers' business plans and strategy and delivering the needed business values.

Reality to Myth #9: -
Wrong. We see there are many rooms for Agile PM. While PMI emphasize on: -

Project Integration Management
Project Scope Management
Project Time Management
Project Cost Management
Project Quality Management
Project Human Resource Management
Project Communications Management
Project Risk Management
Project Procurement Management

In SCRUM/Agile, we adopts lean process, thinking and techniques and we are just focusing on facilitating and empowering the team to help elevate, surface and addressing these issues much sooner and quicker. This would help a qualified project manager to deal/elevate/stay transparent with the risks in the interests of customers, stake holders and the company. Please visit Jesse Fewell's Agile PM.

Reality to Myth #10: -
In SCRUM, we commit to deliver the agreed task items in sprint planning that was put forward together with Product Owner, SCRUM Master and the team. We commit to stay focus and deliver potentially shippable working incremental subsets of solution features. We valued transparency and we work well with fixed cost fixed schedule projects and we collaborate with the customers on business values needed than the scope wanted. Estimates derives from the past velocity remains "estimates" and in short we practiced Agile Principle by working on estimation with commitment and remain transparent, at the same time willing to adapt to the change by encouraging collaboration with customers based on facts collected from the SCRUM artifacts.

Be careful with Linq within Exception Handling Block

2010-10-14T01:07:00.000-07:00

Exception handling try {} catch {} finally {} in .NET always guarantee the finally clause will be executed. However, deferred execution of Linq prove to be able to make a difference, once again thanks to the side effects of linq like I've discussed in the past.

Consider the code block below: -

        protected override IEnumerable<byte[]>  DoReadAsRaw()
        {
         try
         {
          return _settings
           .FieldInfos
           .Select(item => _innerReader.ReadBytes(_innerReader.ReadInt32()));
         }
         catch (EndOfStreamException)
         {
          return null;
         }
        }

This is to read out an Enumerable of byte array. I have another code block below that will call into this method and utilizing the Enumerable of byte array as such: -

         var rawData = DoReadAsRaw();

         return rawData == null ? null : new Record<byte[]>(_settings.FieldInfos, rawData);

Within the Record Constructor, I have the codeblock to iterate thru the enumerable byte array as such: -

        public Record(IEnumerable<FieldInfo> fieldInfos, IEnumerable<T> values)
        {
            if (fieldInfos == null)
            {
                throw new ArgumentNullException("fieldInfos");
            }

            if (values == null)
            {
                throw new ArgumentNullException("values");
            }

            var fieldInfoArray = fieldInfos.ToArray();
            var valueArray = values.ToArray();

            // checks the length of field and data to ensure their length is matching.
            if (fieldInfoArray.Length != valueArray.Length)
            {
                var message = string.Format("Fields and values count mistmatched. Expected {0} fields but was {1}.",
                                            fieldInfoArray.Length, valueArray.Length);
                throw new IndexOutOfRangeException(message);
            }

            _fields = new T[fieldInfoArray.Length];
            _itemNameIndexMap = new Dictionary<string, int>();

            for (var i = 0; i < fieldInfoArray.Length; i++)
            {
                var currentFieldInfo = fieldInfoArray[i];

                _fields[i] = valueArray[i];
                _itemNameIndexMap.Add(currentFieldInfo.Name, i);

                if (!currentFieldInfo.IsKeyField) continue;

                IdentifierFieldIndex = i;
                IdentifierFieldName = currentFieldInfo.Name;
            }
        }

Guess what when unit testing?
EndOfStreamException was not handled!

the past experience of Linq's side effects helps me to quickly identifying that the culprit was the absent of ToArray() or ToList() call right after the linq statement within the try-catch block.

Soon as ToArray() is added into the code, it works beautifully and passed the unit test.

So, best practise for Linq Statement within the try catch block I would recommend is: -

If you are returning anything to the caller after the try catch block, ensure you always force create real collection by calling to ToArray() or ToList().

Thank you.

Test Driven Approach - Positive Impact for Developers - Conclusion

2010-09-27T11:50:00.000-07:00

Going thru this lengthy of workshop series on TDD, I hoped I have covered enough here on positives impact from TDD approach. It takes a developer to practice it to see its worth.

Often in software development, we fall into traps like: -
Lengthy requirement gathering period.
Lengthy conceptualizing and creating over detail blueprint (architecture).
Over design/engineering a software component at early stages.
Lost focus on interactivity between requirement/conceptual model/design phases due to lengthy period.
Less integrability due to disconnected phases.

All these can be nicefully addressed with Agile and TDD.
Agile and TDD are just complementing each other and bringing developers closer to users and hence business.

Final question, why would one insert a break point at several area to debug when a defect was raised? Write a unit test to automate that checks please. Thank you.

Test Driven Approach - Positive Impact for Developers - Workshop IV

2010-09-27T11:39:00.000-07:00

Now we need to implement the Calculate method within Core.Calculator.

So, there are many ways this can be done and whether this method is developed to its best code is not the interest of the workshop. Here is my sample: -

using System;
using System.Collections.Generic;
using System.Linq;

namespace Calculator.Core
{
    public class Calculator
    {
        private const string Add = "+";
        private const string Sub = "-";
        private const string Div = "/";
        private const string Mul = "*";
        private const string Sqrt = "sqrt";
        private const string Percent = "%";
        private const string Reciproc = "reciproc";
        private const string Pow = "pow";

        private readonly string[] _keywords = new []{ Add, Sub, Div, Mul, Sqrt, Percent, Reciproc, Pow, "(", ")", ","};

        private readonly string[] _operatorKeywords = new[] { Add, Sub, Div, Mul, Sqrt, Percent, Reciproc, Pow };

        private static readonly Dictionary<string, Math.MathOperatorHandler> Operators;

        static Calculator()
        {
            Operators = new Dictionary<string, Math.MathOperatorHandler>
                             {
                                 {Add, Math.Add},
                                 {Reciproc, Math.Reciproc},
                                 {Pow, Math.Pow},
                                 {Sub, Math.Subtract},
                                 {Div, Math.Divide},
                                 {Mul, Math.Multiply},
                                 {Sqrt, Math.SquareRoot},
                                 {Percent, Math.Percent}
                             };
        }

        public string Calculate(string formula)
        {
            double temp;
            if (double.TryParse(formula, out temp))
            {
                return formula;
            }
            var values = formula.Split(_keywords, StringSplitOptions.RemoveEmptyEntries);
            if (values.Length == 0)
            {
                var msg = string.Format("No values to be calculated. Values length is zero for formula {0}.", formula);
                throw new UnexpectedCalculationException(msg);
            }

            var subPart = formula.TrimStart(values[0].ToCharArray());
            var opCode = _operatorKeywords.FirstOrDefault(subPart.StartsWith);

            if (string.IsNullOrEmpty(opCode))
            {
                var msg = string.Format("No calculation operator found in formula {0}.", formula);
                throw new UnexpectedCalculationException(msg);
            }

            Math.MathOperatorHandler mo;

            // this is only possible when there is a %.
            if (subPart.EndsWith(Percent))
            {
                if (!Operators.TryGetValue(Percent, out mo))
                {
                    var msg = string.Format("Cannot resolve calculation operator of {0}.", Percent);
                    throw new UnexpectedCalculationException(msg);
                }
                values[values.Length - 1] = mo(values.Select(item => double.Parse(item))).ToString();
            }

            if (!Operators.TryGetValue(opCode, out mo))
            {
                var msg = string.Format("Cannot resolve calculation operator of {0}.", opCode);
                throw new UnexpectedCalculationException(msg);
            }
            var valueArray = values.Select(item => double.Parse(item));
            var result = mo(valueArray);
            return result.ToString();
        }
    }
}

But what is more interesting is at the supporting class below: -

    internal static class Math
    {
        private delegate T TryExecuteHandler<out T>();

        internal delegate double MathOperatorHandler(IEnumerable<double> values);

        private static double Handle(TryExecuteHandler<double> handler)
        {
            try
            {
                return handler();
            }
            catch(FormatException e)
            {
                throw new IncompatibleDataTypeFormulaException("Unexpected data format.", e);
            }
        }

        internal static double Add(IEnumerable<double> values)
        {
            return Handle(() => values.Sum());
        }

        internal static double Subtract(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  var result = valueArray[0];
                                  for (var i = 1; i < valueArray.Length; i++)
                                  {
                                      result -= valueArray[i];
                                  }
                                  return result;
                              });
        }

        internal static double Multiply(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  var result = valueArray[0];
                                  for (var i = 1; i < valueArray.Length; i++)
                                  {
                                      result *= valueArray[i];
                                  }
                                  return result;
                              });
        }

        internal static double Divide(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  var result = valueArray[0];
                                  for (var i = 1; i < valueArray.Length; i++)
                                  {
                                      result /= valueArray[i];
                                  }
                                  return result;
                              });
        }

        internal static double SquareRoot(IEnumerable<double> values)
        {
            return Handle(() =>
                              {

                                  var valueArray = values.ToArray();
                                  if (valueArray.Length > 1)
                                  {
                                      var msg = string.Format("Expect valueArray of length 1 but was {0}.", valueArray.Length);
                                      throw new UnexpectedCalculationException(msg);
                                  }

                                  return System.Math.Sqrt(valueArray[0]);
                              });
        }

        internal static double Pow(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  var result = valueArray[0];
                                  for (var i = 1; i < valueArray.Length; i++)
                                  {
                                      result = System.Math.Pow(result, valueArray[i]);
                                  }
                                  return result;
                              });
        }

        internal static double Reciproc(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  if (valueArray.Length > 1)
                                  {
                                      var msg = string.Format("Expect valueArray of length 1 but was {0}.", valueArray.Length);
                                      throw new UnexpectedCalculationException(msg);
                                  }

                                  return 1/valueArray[0];
                              });
        }

        internal static double Percent(IEnumerable<double> values)
        {
            return Handle(() =>
                              {
                                  var valueArray = values.ToArray();
                                  if (valueArray.Length != 2)
                                  {
                                      var msg = string.Format("Expect valueArray of length 2 but was {0}.", valueArray.Length);
                                      throw new UnexpectedCalculationException(msg);
                                  }
                                  return valueArray[0]*valueArray[1]/100;
                              });
        }
    }

In the unit tests, we expected two different kinds of exceptions as such: -

    [Serializable]
    public class IncompatibleDataTypeFormulaException : Exception
    {
        public IncompatibleDataTypeFormulaException(string message, Exception innerException)
            :base(message, innerException)
        {
        }
    }

    /// <summary>
    /// <see cref="UnexpectedCalculationException"/> - Exception indicates that there were some unexpected internal calculation error.
    /// </summary>
    [Serializable]
    public class UnexpectedCalculationException:ApplicationException
    {
        public UnexpectedCalculationException(string message)
            :base(message)
        {
        }
    }

Now that I don't want to make the case complicated by implementing a FormulaParser and FormulaToken all that sorts just for a simple calculator, I opted for the static class implementation named Math. This supporting class is never in my mind that will be shared. It is my internal calculation logic that is for abstraction. It has very high chances that I want improve it in the future.

This class should never be unit tested.

If you do, you will quickly find you do owe yourself for a blame. Unit testing your abstraction logic will pressurize on your responsibility of maintaining highest potential to change API, process workflow, class design and etc. You ended do a lot of work and then undone many work that you were spending day and night maintaining it.

Best practice of TDD said, let those highly changeable code (abstraction logic) automatically covered by just unit testing your public API. If it was not covered automatically, you have written unplanned code and that is the risk for defects or bugs. You will be helped by the coverage to uncover the statistic for this area of risk to you.

Note: The only place that is not well-tested is at UI.

Next, look at the abstraction logic again, there is a pattern of repeatedly throwing exception is needed for most of the function like Add, Divide, Multiply etc, this just comes naturally that I will need some aspect oriented programming for exception handling in this function. Hey, now I need to use AOP design patterns with Code Injection facility.

This is awesomely beautiful isn't it, doing it TDD?

Test Driven Approach - Positive Impact for Developers - Workshop III (Part B)

2010-09-27T11:09:00.000-07:00

Now, lets us try to compile and run the test again. As expected, it won't compile at all. We need refactoring to the unit tests at Calculator.Tests as such: -

using Calculator.Core;
using Calculator.Presenter;
using NUnit.Framework;

namespace Calculator.Tests
{
    public class CalculatorMockView : ICalculatorView
    {
        public string Result { get; set; }
        public string Formula { get; set; }
    }
    
    [TestFixture]
    public class CalculatorTester
    {
        private Presenter.Calculator _calculator;
        private ICalculatorView _view;

        [SetUp]
        public void SetUp()
        {
            _view = new CalculatorMockView();
            _calculator = new Presenter.Calculator(_view);
        }

        [TestCase("6", "6")]
        [TestCase("5 + 6", "11")]
        [TestCase("3 - 2", "1" )]
        [TestCase("5 * 6", "30")]
        [TestCase("12 / 6", "2")]
        [TestCase("5 + 6%", "5.3")]
        [TestCase("reciproc(2)", "0.5")]
        [TestCase("pow(2,3)", "8")]
        [TestCase("sqrt(4)", "2")]
        [TestCase("12 / 6 / 8", "0.25")]
        [TestCase("reciproc(2,3,2)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("sqrt(4, 21)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("A + B + C", "", ExpectedException = typeof(IncompatibleDataTypeFormulaException))]
        [TestCase("HelloWorld(2,3,2)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("2 3 2", "", ExpectedException = typeof(UnexpectedCalculationException))]
        public void EvaluationTest(string formula, string expected)
        {
            _view.Formula = formula;
            _calculator.Evaluate();
            Assert.AreEqual(_view.Result, expected);
        }
    }
}

And Calculator.Core.Tests as such: -

using NUnit.Framework;

namespace Calculator.Core.Tests
{
    [TestFixture]
    public class CalculatorTests
    {
        readonly Calculator _calculator = new Calculator();

        [TestCase("6", "6")]
        [TestCase("5 + 6", "11")]
        [TestCase("3 - 2", "1")]
        [TestCase("5 * 6", "30")]
        [TestCase("12 / 6", "2")]
        [TestCase("5 + 6%", "5.3")]
        [TestCase("reciproc(2)", "0.5")]
        [TestCase("pow(2,3)", "8")]
        [TestCase("sqrt(4)", "2")]
        [TestCase("12 / 6 / 8", "0.25")]
        [TestCase("reciproc(2,3,2)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("sqrt(4, 21)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("A + B + C", "", ExpectedException = typeof(IncompatibleDataTypeFormulaException))]
        [TestCase("HelloWorld(2,3,2)", "", ExpectedException = typeof(UnexpectedCalculationException))]
        [TestCase("2 3 2", "", ExpectedException = typeof(UnexpectedCalculationException))]
        public void EvaluationTest(string formula, string expected)
        {
            Assert.AreEqual(expected, _calculator.Calculate(formula));
        }
    }
}

Above, I have also added a few additional test cases where necessary. Now lets us compile and test run them. As expected, they are still failed.

Before we proceed, to correct all these tests, let us revisit what have we done so far?
1. We created solution based on some (small) fundamental architecture design.
2. We created a wire frame demonstrated how this look like conceptually.
3. We then created some unit tests focuses on "how we think the public API would be useful and clean". (API Usability Concern from consumer perspective.)
4. We introduce new projects (core) to ensure we provide better separation of business core logic. (Separation of Concern and Single Responsibility)
5. We then decide again how is the public API of the new project just added would look like. (API Usability again)
6. We implemented the presenter logic, wire up the UI wire frame, and the Core logic classes. (some UI design patterns were introduced for unit tests and better code separation).
7. We refactor the some API within its implementation and its corresponding unit testings in Calculator.Tests and Calculator.Core.Tests. (Refactoring based on needs).

Now, all the above just came naturally as we do our development in TDD fashion.

So, the next workshop, we will be aiming to clear all these unit tests.

Test Driven Approach - Positive Impact for Developers - Workshop III (Part A)

2010-09-27T10:44:00.000-07:00

Quickly, we are now at 3rd session. Before we start, I would like to cover a bit on the missing architecture detail. This small application will be (hopefully) tailored into 3 layers design. UI, Core Logic, Data Access (if any). It is not that the requirements asking for so, it is simply some thought to have clear separation such that each component can be more tidy up with better cut of responsibility. Also, I hope I will have an opportunity in the future to bring this example into some further discussion on Scaling Out and Services Oriented fashion.

We have not create any data access layer assembly, but because we created a new assembly, "Core", hence a new Unit Test project for Core project is needed.

Enough said, now lets focus on the assignment. In reality, the calculator will be very unlikely to perform the calculation of such formula at once, "5 + 5% * 50 / 2 + pow(2,3)". Soon as you type in "5 + 5%", the calculator has already responded with the result. (Please verify with Windows Calculator). So, we will probably not cover this within this series of workshop.

Question: Where do we start coding to pass the test?
Answer: Let's start with the UI.

Question: How do we test the code in the UI?
Answer: We need a presenter or controller logic.

Calculator Presenter class develop as such: -

using System;
using System.Text;

namespace Calculator.Presenter
{
    public class Calculator
    {
        private readonly ICalculatorView _view;
        private readonly Core.Calculator _calculatorCore;

        public Calculator(ICalculatorView view)
        {
            _view = view;
            _calculatorCore = new Core.Calculator();
        }

        public void Evaluate()
        {
            _view.Result = _calculatorCore.Calculate(_view.Formula);
        }
    }

    public interface ICalculatorView
    {
        string Result { get; set; }
        string Formula { get; set; }
    }
}

This will need an additional Calculator class within the Core assembly as such: -

    public class Calculator
    {
        public string Calculate(string formula)
        {
            throw new NotImplementedException();
        }
    }

Now we sort of have everything (API) within this application that needs public access modifier done. We need to ensure the new public API sits under the Core project is unit tested as well. In this case, it is very similar to the earlier unit test created in Workshop II.

Next, we focus on how to wire up the UI into the presenter logic. This is what generated by VS 2010.

namespace Calculator
{
    static class Program
    {
        /// <summary>
        /// The main entry point for the application.
        /// </summary>
        [STAThread]
        static void Main()
        {
            Application.EnableVisualStyles();
            Application.SetCompatibleTextRenderingDefault(false);

            Application.Run(new CalculatorForm());
        }
    }
}

And here, we improve the sample WinForm class created during prototyping to utilize the presenter coded, as such: -

using System.Windows.Forms;
using Calculator.Presenter;

namespace Calculator
{
    public partial class CalculatorForm : Form, ICalculatorView
    {
        private Presenter.Calculator _calculator;

        public CalculatorForm()
        {
            InitializeComponent();
            _calculator = new Presenter.Calculator(this);
        }

        public string Result
        {
            get { return textBoxDisplay.Text; }
            set { textBoxDisplay.Text = value; }
        }

        public string Formula
        {
            get { return textBoxDisplay.Text; }
            set
            {
                textBoxDisplay.Text = value;
                textBoxDisplay.Refresh();
            }
        }
    }
}

Till this stage, we already have wire up most of the API needed within this application.

Test Driven Approach - Positive Impact for Developers - Workshop II

2010-09-26T08:26:00.000-07:00

First of all, there were couple of useful tools for Test Driven within Visual Studio .NET 2008 or 2010.

These are the tools the I usually use: -
1. NUnit
2. TestDriven.NET
3. JetBrains ReSharper

Note: JetBrains resharper is not free though.

So, after all, how should we start the development based on the requirements?

I usually create a new Test Project first, named Calculator.Tests.

For any TDD development, we should always attempt to write the tests from highest level possible with the simplest code. To achieve this, we should always attempt with the simplest API that we are going to deal with.

So, I would imagine the calculator class to work like this: -

Calculator cal = new Calculator();
var result = cal.Evaluate(formula);

Hence, I created a class within the Calculator.Tests as such: -

using NUnit.Framework;

namespace Calculator.Tests
{
    [TestFixture]
    public class CalculatorTester
    {
        private Calculator _calculator;

        [SetUp]
        public void SetUp()
        {
            _calculator = new Calculator(); 
        }

        [TestCase("5 + 6", "11")]
        [TestCase("3 - 2", "1")]
        [TestCase("5 * 6", "30")]
        [TestCase("12 / 6", "2")]
        [TestCase("2 * 3 - 4 / 5 + 6", "11.2")]
        [TestCase("5 + 6%", "5.3")]
        [TestCase("reciproc(2)", "0.5")]
        [TestCase("pow(2,3)", "8")]
        [TestCase(new [] {"A + B + C", string.Empty},
                    "Incompatible data type evaluation test.",
                    typeof(IncompatibleDataTypeFormulaException))]
        public void EvaluationTest(string formula, string expected)
        {
            var result = _calculator.Evaluate(formula);
            Assert.AreEqual(result.ToString(), expected);
        }
    }
}

Now with have almost demonstrated most of the features needed using various test cases based on TestCaseAttribute of NUnit 2.5.

You may be asking, so, this is the business logic, where is the UI testing? Isn't it we must start from the UI?

Yes, you got it half spots on. We must start from the UI (highest level), however are you sure this calculator class is just a business logic class? We shall see then.

Now, lets try to fix all the compilation error by creating some fundamental skeleton for it to compile.

public class Calculator
{
    public double Evaluate(string formula)
    {
        throw new NotImplementedException();
    }
}

public class IncompatibleDataTypeFormulaException:Exception
{
}

Now that we have written the unit tests and the compilation is done. We should test run it and see all of them failed.

As expected, all test cases failed. I would be extremely surprised if any of this passed. :)

So, the next assignment is to get them green.

Test Driven Approach - Positive Impact for Developers - Workshop I

2010-09-24T04:02:00.001-07:00

Start from this post and over the next few posts, I will be covering the Test Driven Approach - Positive Impacts for Developers.

Impacts: -
1. Think like a user
2. Clean APIs Design
3. Improves Abstractions
4. Better Used of Patterns
5. Improves Separation of Concerns.

Let starts with some fundamental user requirements on a calculator.

I can do fundamental calculation, +, -, *, /, sqrt, %, pow and 1/x.
I can use whole numbers or rational numbers as the input.
I can defined the result format within the format of whole number or rational number.
I can specify the number of decimal places for the results.
I can define a formula and save it.
I can load a formula and calculate using the loaded formula.
I can only specify the values for variables within the formula.

There is all we have for the requirements.

So, if we're to start right away, we should have start writing the unit tests first.

A quick glance into the requirements, it seems quite straight forward to accomplish.

Questions: -
1. How many of you already start thinking of classes for logic separation, like Adder, Subtractor, Divisor and Multiplicator etc.?

2. How many of you already start writing unit tests right away based on the requirement defined?

"If there is anyone falling into those traps, you're wrong!" The requirement wasn't quite defined into a stage where this can be started.

Based on the requirements given, we should have at least asking how would it be deployed? Web Services, Windows Services, Web Apps or Windows Forms Apps for instance.

So, to make it more simple, this is a Win Form application, just like the calculator in Windows.

New User Requirements: -
"I can use it like many windows desktop application."

So now, we are good to start, however, not unit testing yet. It is the UI wireframe (prototyping for GUI, i.e. Forms.) Here it is: -

Now, we can start writing some tests, that will be in workshop II.

ETL Sorting or SQL Sorting (ORDER BY)

2010-09-08T09:51:00.001-07:00

ETL Magic

Often hear about ETL (Extract, Transform, Load) solution for better data integration between applications, systems or solution within an enterprise. What are the major consideration before choosing an appropriate ETL solution?

Sorting, Merging, Unification, Transformation and Loading are quite commonly happen within Data Integration stage. Often, we heard of NSort, CoSort (IRI), SyncSort and others famous 3rd party ETL solution and tools. Each of these offers some great performance if not exceptional in their respective technical strength with many data integration efficiency (Terabytes of data size) bar being set higher and higher.

EnTerprise solution missing Link?

Quite honestly, many solution architects would think of the use in these ready made ETL solution to glue data movement processes within the sub-solutions altogether to not just making data extraction, transformation and loading much faster (conceptually), but also act as a missing link to gel many independent components within the enterprise solution to ensure they get the right data in timely fashion before going live.

Question of the day

What would be the topic of discussion today is rather, will those 3rd parties solution offers the best performance value to the enterprise solution and often considered as the missing link in the solutions integration?

ETL Rules of thumb: -

If data is already in the database and you would want to extract all these data into another database, consider doing the sorting and filtering related tasks within your database server itself. Ordering, filtering and unification tasks within the database server is always consider as "as good" when compared to the other ETL solution.

If the data has to be somewhat inserted into tables within database, consider doing sorting, unification, merging (joins), filtering, data enrichment (meta tagging) and etc. altogether within the database server. Again, the same principles applied here. Database server can handle all those as good as the third parties ETL.

If the batch data integration process can be avoided by putting it into an incremental data integration approach, by all means doing it in an incremental fashion with proper application/data partitioning.

If the data is migrating from one database platform into another, try to look for some out of the box integration solutions. For example, SSIS for Microsoft SQL Server. (SSIS is free and has also recently breaking some integration records.)

Never ever thinking of re-inventing your own ETL solution, from ground up, if you are not intending to step into the red ocean, ETL business.

Consider 3rd parties ETL solutions when the data source is originated from a non-database model and the data target is another non-database model.

Consider 3rd parties ETL solutions when majority of the data transformation, extraction, unification, merging, filtering and sorting involves many different types of data sources.

Consider 3rd parties ETL solutions when it has more than one intended data target that are of different platforms.

Take Away

SQL ordering, filtering, conditional relationship/merging (joins) are almost equally as fast as the 3rd parties ETL tools.

No real magics with ready made ETL solutions, it is the solution integration architecture that counts.

Fancy solutions will cost more in terms of investment capital, learning curve, maintenance efforts, business opportunity, end product selling prices (to your customers).

Side Effects of Linq Select Statements due to Deferred Execution

2010-08-19T19:15:00.000-07:00

I have a function to read data from a delimited text that is splitted into a string array. To the caller didn’t specifies the columns to extract, it extract all columns, and only extracts the specified one otherwise.

Assumed that I have a delimited text that contains 2 columns only: -

public DataField[] Read()
{
    // splits the text according to the defined delimiters.
    var data = _reader.Read();
    // when no columns specified, take all column from data.
    if (_columns == null)
    {
        var counter = 0;
        // save as the instance fields to avoid recompilation.
        _columns = data.Select(item => new ColumnInfo
                                        {
                                            Index = counter++
                                        });
    }

    return data == null
    ? null
    : _columns.Select(item => new DataField
                                {
                                    Column = item,
                                    Data = data[item.Index]
                                }).ToArray();
}

To my surprise that this Read method keeps throwing ArrayOutOfBoundException after the first attempt. Can anyone guess what’s wrong? :) (Answer: scroll down please)

After the first call, the enumerator is wellover (at its end element). However, IEnumerable created (which I thought it was) from a Linq statement is not a real creation of IEnumerable instance.

_columns = data.Select(item => new ColumnInfo
                                {
                                    Index = counter++
                                });

This statement is rather associating _columns field to the Linq statement (like a callback, I think). The subsequent Read() call would continue to invoke the lambda expression within the linq statement for next element, which created column with Index=3. (This also explains why linq’s performance is good without much overhead).

To workaround this side effets, one need to really tell the Linq function to create a real array or list for caching and the side effect will be gone.
Solution as such: -

public DataField[] Read()
{
    var data = _reader.Read();

    if (_columns == null)
    {
        var counter = 0;
        // ToArray() will tear off the callback by caching a real array, not callback.
        _columns = data.Select(item => new ColumnInfo
                                        {
                                            Index = counter++
                                        }).ToArray();
    }

    return data == null
        ? null
        : _columns.Select(item => new DataField
        {
            Column = item,
            Data = data[item.Index]
        }).ToArray();
}

Read Jon Skeet article on Human Linq

Also here about side effects with select

TDD: Only Unit Testing Public

2010-08-05T00:33:00.000-07:00

Many times when I spoke to people about TDD and Unit Testing, quite surprisingly I will be trapped into a debate of "Unit Test Public only?".

My quick answer is "Yes".

Reason?
Internal, Protected, Private "properties" of an Assembly or Class are the abstraction for each of them respectively. Unit tests should never cross the border to break the abstraction intend of a design.

DISCLAIMER: "properties" I used here refers to methods, subtypes, property (fields) etc.

Come to think of this topic again, why would one asked the question of such like? (unit test public?)
My only view is that, these may be the people who are writing real implementation first before writing the tests. They knew the design and implementation before even writing unit test and they knew the importance of the internal implementation and they would like to secure it (secure logic implementation intend). Are these suggesting TDD practice? (Refer to my earlier TDD post HERE).

If one started to think of the application (usage) of code to be developed first before thinking how to develop it (or design it), one should start with the accessible content. (Which is public content). Get the usage right is the prime criteria for TDD and also many software quality assurance measure. From the righteous usage of the public "properties" corresponded with right expectation and that makes the "right" deliverable.

So, spend more time thinking of what may be "right" for the time of writing, and ensure unit tested the "right" intend that you would want to enforced. Internal implementation are abstraction that meant for future re-development or refactoring or enhancement that usually and very likely to change from time to time without prior noticed.

This way, you have done your team and yourself a favor by created some maintainable unit tests.

MSSQL Common Table Expression (CTE)

2010-08-03T03:55:00.000-07:00

What is Common Table Expression?

Can be thought as a temporally result set that is defined within the SQL execution scope.
Usage: -

Create Recursive Query
Substitute for a view
Enable Grouping by column that is derived from a scalar sub select.
Referencing the resulting table multiple times.
Available since SQL Server 2005

Syntax: -

WITH [(coloumn name [, column name...n])]
AS
(
/* anchor member definition */
UNION ALL | UNION | EXCEPT | INTERCECT
/* recursion member definition*/
)

Statement to use with CTE include INSERT, UPDATE, DELETE, CREATE VIEW.

CTE can be used without recursion member and that will serve very well as the temporally result set for further joining.

Example: -

/*************** BEGIN OF QUERY **********************/
USE NRule
GO

WITH DocumentReport
AS
(
SELECT a.id, a.[Description], a.Name AS [DocumentName], c.Name AS [AttributeName], d.value
FROM Documents AS a
INNER JOIN DocumentMetaAttributePathIndex AS b
ON a.id = b.documentId
INNER JOIN MetaAttributes AS c
ON b.attributeId = c.id
INNER JOIN AttributeValues AS d
ON b.valueId = d.id
)
SELECT *
FROM DocumentReport ORDER BY id
GO
/*************** END OF QUERY **********************/

CTE would be best use to query of outcome that is expanding dynamically. Typical example is to trace the tree or graph data structure represented under adjancent list table design pattern within relational model of a database.

Example: -

/*************** BEGIN OF QUERY **********************/
USE NRule
GO

DECLARE @documentId bigint
DECLARE @ancestorId bigint
SET @documentId = 1
SET @ancestorId = NULL;
WITH AttributeHierarchyReport(AncestorId, AttributeId, DocumentId, ValueId, LEVEL)
AS
(
-- Anchor member definition
SELECT meta.AncestorId, meta.attributeId, meta.[DocumentId], meta.[ValueId], 0 AS LEVEL
FROM dbo.DocumentMetaAttributePathIndex AS meta
WHERE (@ancestorId is NULL or meta.AncestorId = @ancestorId) AND
(@documentId is NULL or meta.DocumentId = @documentId)
UNION ALL
-- Recursive member definition
SELECT meta.ancestorId, meta.attributeId, meta.[documentId], meta.[valueId], LEVEL + 1
FROM dbo.DocumentMetaAttributePathIndex AS meta
INNER JOIN AttributeHierarchyReport rpt ON meta.ancestorId = rpt.AttributeId
WHERE (@ancestorId is NULL or meta.AncestorId = @ancestorId) AND
(@documentId is NULL or meta.DocumentId = @documentId)
)
SELECT AncestorId, AttributeId, DocumentId, ValueId, Level
FROM AttributeHierarchyReport ORDER BY Level;
GO
/*************** END OF QUERY **********************/

However, watch out for the data redundancy that is appearing in the query result set. Redundant query result set within the recursive CTE proven to be a killer to your database application using it. It grows the result set sometimes exponentially until it kills the application by running out of memory.

Tweak is needed to CTE when redundant row sets happened and if you care about your application, never live with redundant row sets by either further tuning your CTE or improve your Database/Table design to help better/efficient use with your CTE.

Example: -

/*************** BEGIN OF QUERY **********************/
/* this version is using additional column ParentDescendantId to enforce root node validation at anchor member */
USE NRule
GO

DECLARE @documentId bigint
DECLARE @ancestorId bigint
/* @documentId = NULL would mean to get all attributes for all documents regardlessly */
/* alternatively we can use the @documentId to narrow down to a specific document attribute tree for retrieval */
SET @documentId = 1
/* @ancestorId = NULL would mean to get all attributes regardlessly */
/* alternatively we can use the ancestorId to narrow down the attribute tree for retrieval */
SET @ancestorId = NULL;
WITH AttributeHierarchyReport(AncestorId, DescendantId, DocumentId, ValueId, ParentDescendantId, LEVEL)
AS
(
-- Anchor member definition
SELECT meta.AncestorId, meta.attributeId, meta.[DocumentId], meta.[ValueId], parent.attributeId, 1 AS LEVEL
FROM dbo.DocumentMetaAttributePathIndex AS meta
LEFT JOIN dbo.DocumentMetaAttributePathIndex AS parent ON meta.AncestorId = parent.attributeId
WHERE (@ancestorId is NULL or meta.AncestorId = @ancestorId) AND /* Ensure the ancestor id is matched */
    (@documentId is NULL or meta.DocumentId = @documentId) AND /* Ensure the document id is matched */
    (parent.attributeId IS NULL) /* Ensure there is no ancestor, Root node only */
UNION ALL
-- Recursive member definition
SELECT meta.ancestorId, meta.attributeId, meta.[documentId], meta.[valueId], parent.DescendantId, LEVEL + 1
FROM dbo.DocumentMetaAttributePathIndex AS meta
INNER JOIN AttributeHierarchyReport parent ON meta.ancestorId = parent.DescendantId /* Ensure all is child is recursively joined */
)
SELECT AncestorId, DescendantId, DocumentId, ValueId, ParentDescendantId, Level
FROM AttributeHierarchyReport;
GO
/*************** END OF QUERY **********************/

Don't Limiting Yourself Much Than Your Users or Clients Do.

2010-08-01T21:41:00.000-07:00

Many times, before planning on a new software release you often hear the voice of backward compatibility, upgrade costs, additional component dependencies, operating platforms upgrades and etc. from the team. Are these important? Yes they do. Are these debates necessary before the plan of a release? Absolutely not. We need to look at it and aware of it, but lets not debating it now.

A new software release is to ensure it maintains attractiveness and sounds of improve workability, usability, flexibility, integrativity, scalability, security, martketability and many more.

Before one starting to look at the release plan or product backlog, and if you have already be the first to attack on your own team, you are not doing your team a favor, but holding them back to be closer with FAILURE and not able to deliver.

One phrase that I often hear and it is always true is that "Many times, we often limiting ourselves much more than the customers or users do."

Wake up guys! Be wild and creative when innovating of what you want to do to your software during visioning stage, and only come to be more focus and cautious for not over committing when you are planning and delivering it.

Overly conservative does no help of bringing innovations. Bottom line, don't limiting yourself much than your users or customers do would always be a golden phrase to follow.

Test Driven Development

2010-07-22T04:35:00.000-07:00

Five key principle questions must be asked all time:

What Do I Want to Accomplish?
Does It Do What I Want?
Does It Do What I Want All of the Time?
Can I Depend On It?
Does It Document My Intent?

TDD Process Itinerary

Obtains Project Specification Documents to identify specification needs, usually the action from the UML CASE diagram. (What needs to be done?)
Write the test procedures and test methods (Unit Test) to enforce it achieve the specification needs as how it was documented in Project Specification Document. (Does it do what I want?)
Write the code just to pass the test.
Ensure and maintain the tests from never fail at all times. (Does it do what I want at all time and Can I depend on it?)
Re-factor the code for readability and maintainability, while ensure the tests remain intact to the specification. (Does it document my intent?)

Few ways to achieve unit testing: -

Dummy
Fakes
Mocks
Stubs

To be continue...

Top 10 Unwanted Characteristic of a Developer

2010-07-22T03:30:00.000-07:00

1. Egoness
Example: -
We are senior and we have many years of exposure and we know what we need to do, please keep your explanation to yourself.
Outcome: -
Fail to grow as a team, fail to deliver or deliver with ULTRA high defects. Copying design ideas from others but not bothering to ask and resulting in sub-standard code, but proclaiming that is the beauty of it when challenged.

2. Year-istic Mathematician
Example: -
I want a lead role because of my many years of exposure. I work with many big profound companies and I derserved the lead role.
Outcome: -
Not able to justify the pros and cons and delivering sub standard work or not delivering at all.

3. Favor Quick Wins over Quality
Example: -
Deliver the max within the shortest time is the primary concern of the company to make money. Patterns and design are purism and that often complicates the development.
Outcome: -
Fails to grow as a team or individually. Produced sub standard codes that needs major rework whenever enhancement needs imposed.

4. Never Admit Own Mistakes
Example: -
The bugs had happened because the calling function never do a parameter validation and error checking.
The late delivery was largely due to inconsistent and or incomplete requirements.
Outcome: -
Never learn from mistakes and similar problems reoccuring over and over again. It is quite clear that if it is not your problem but it somewhat impact your work delivery or team goals, it is everyone's problem and deal with it if you can, escalates otherwise.
To be continue....

.NET Application Slow?

2010-07-22T03:00:00.000-07:00

Many people complained about .NET application being slow. First question first, do you know how .NET work internally?

Pickup a good book like CLR via C# (Jeffrey Ricther) or Framework Design Guideline (Cwalina, Abrams) would help.

If you don't understand how GC (Garbage Collector) works, please do yourself a favor.

If you find yourself writing lots of codes (and redundant codes), do yourself a favor reading Design Patterns.

If you find yourself adding lots of project/assembly references into a project, do yourself a favor reading books about SoC (Separation of Concern or Single Responsibility or Architectural Consideration).

If you find yourself fixing lots of bugs that subsequently introduce more and more bugs (defects), change your development engineering process but adopting TDD (Test Driven Development) approach. (Unit Testing)

I'm hoping to have touch more indepth details of these topics in the coming post.