Crossover from orthogonal to unitary symmetry for ballistic electron transport in chaotic nicrostructures