If quagga is not installed look at the installation description below.
To kick start zebra + ospfd one should perform:

- create virtual interface vni0:
ifconfig vni0 plumb
ifconfig vni0 x.x.x.x netmask 255.255.255.255

- create configuration files /etc/quagga/zebra.conf and /etc/quagga/ospfd.conf, for example
/etc/quagga/zebra.conf:

!
! Zebra configuration saved from vty
! 2013/03/14 18:52:16
!
hostname Router
password zzz
enable password zzz
log file /var/tmp/zebra.log
!
interface bge0
!
interface bge1
!
interface lo0
!
interface vni0
!
ip prefix-list ospf seq 10 permit 10.0.0.0/8 le 32
!
route-map DummySRC permit 10
 match ip address prefix-list ospf
 set src 10.1.1.1
!
!
ip protocol ospf route-map DummySRC
!
line vty
!


/etc/quagga/ospfd.conf:
!
! Zebra configuration saved from vty
! 2013/03/18 16:45:22
!
hostname ospfd
password zzz
enable password zzz
!
!
!
interface bge0
!
interface bge1
!
interface lo0
!
interface vni0
!
router ospf
 ospf router-id 10.1.1.1
 passive-interface vni0
 network 10.1.1.1/32 area 3.3.3.3
 network 172.16.2.0/24 area 3.3.3.3
 network 172.16.3.0/24 area 3.3.3.3
 area 3.3.3.3 stub
!
line vty
!

- enable telnet management interfaces for zebra and ospfd daemons:
routeadm -l zebra
svc:/network/routing/zebra:quagga:
 batch = false
 group =
 pid_file =
 stability = Evolving
 user =
 value_authorization = solaris.smf.value.routing
 vty_address = 127.0.0.1
 config_file =
 vty_port = 2601

routeadm -m zebra:quagga vty_address="127.0.0.1"
routeadm -m zebra:quagga vty_port="2601"
routeadm -u

routeadm -m ospf:quagga vty_address="127.0.0.1"
routeadm -m ospf:quagga vty_port="2604"
routeadm -u

- start services:
routeadm -s routing-svcs=ospf:quagga -e ipv4-routing -u

routeadm
 Configuration Current Current
 Option Configuration System State
---------------------------------------------------------------
 IPv4 routing enabled enabled
 IPv6 routing disabled disabled
 IPv4 forwarding disabled disabled
 IPv6 forwarding disabled disabled

 Routing services "ospf:quagga"

Routing daemons:

 STATE FMRI
 disabled svc:/network/routing/legacy-routing:ipv4
 disabled svc:/network/routing/legacy-routing:ipv6
 disabled svc:/network/routing/ndp:default
 disabled svc:/network/routing/rdisc:default
 disabled svc:/network/routing/ripng:default
 disabled svc:/network/routing/ripng:quagga
 disabled svc:/network/routing/route:default
 online svc:/network/routing/zebra:quagga
 disabled svc:/network/routing/rip:quagga
 online svc:/network/routing/ospf:quagga
 disabled svc:/network/routing/ospf6:quagga
 disabled svc:/network/routing/bgp:quagga



On default quagga for Solaris 10 is built without vtysh support.
To fix the issue one could rebuild the package from sources:
> ./configure --localstatedir=/var/run/quagga --enable-gcc-rdynamic --enable-opaque-lsa --enable-ospf-te --enable-multipath=64 --enable-user=root --enable-ospf-te --enable-ospfclient=yes --enable-ospfapi=yes --enable-group=root --enable-nssa --enable-opaque-lsa --enable-vtysh
> gmake
# make install

#/usr/local/bin/vtysh -d zebra

to test service startup:
/usr/sbin/zebra -P 2601 -A 127.0.0.1 --pid_file /var/run/quagga/zebra.pid
/usr/sbin/ospfd -P 2604 -A 127.0.0.1 --pid_file /var/run/quagga/ospfd.pid

port and address in the parameters do not guarantee that the processes will listen on the port

http://quagga.piotrkosoft.net/download/binaries/solaris/README.pkg-0.98.4-0.99.0.txt

Solaris 10 Quagga Packages.
---------------------------

These are beta packages of Quagga for Solaris, with SMF support. 

 These packages were compiled for Solaris 10 and will attempt to install SMF
specific files, however the binaries themselves may (or may not) work on
Solaris 8 and 9. Please read all of the notes below before attempting to use
these binaries.

These binaries were built with gcc with the following configure options:

./configure --prefix=/usr/local/quagga --localstatedir=/var/run/quagga
--enable-gcc-rdynamic --enable-opaque-lsa --enable-ospf-te
--enable-multipath=64 --enable-user=quagga --enable-ospf-te
--enable-ospfclient=yes --enable-ospfapi=yes --enable-group=quagga
--enable-nssa --enable-opaque-lsa

This means that by default:

- The daemons will try switch credentials to the 'quagga' user and group.
 These user and group IDs do not exist per default on Solaris, you will
 have to either create this user and group, **before** installing the
 packages OR otherwise change Quagga/user and Quagga/group properties for
 each service after installing the packages and verify the ownership and
 permission on /var/run/quagga.

- The daemons will try create their UNIX sockets in /var/run/quagga. This
 directory does not exist per default on Solaris, but will be created by
 the package.

 It must be writable to the user the daemons will effectively run
 as, eg if you created a quagga user and run the daemons as such, then
 /var/run/quagga should be owned by quagga and mode 0611. The packages will
 take care of this provided the appropriate user/group existed on the
 system.

- The daemons look to /usr/local/quagga/etc/<name>.conf for their config
 file (which must be readable to the user the daemon is run as). See the
 'Quagga/config_file' SMF property for each service instance on how to
 configure this.

Further notes:

- zebra does not yet support alias address interfaces on Solaris. Do not
 expect zebra to correctly function if you have alias interfaces
 configured. As a work around, one can plumb 'vni' interfaces and use these
 for alias addresses, one vniX interface per address. As vni interfaces are
 specific to S10, this workaround is specific to S10 too - alias interfaces
 hence are not supported by zebra on Solaris 9 or earlier. This will
 hopefully be remedied in a future release of Quagga.

- These binaries are not supported by Sun Microsystems in any way. Please
 use the normal Quagga channels for support and/or bug reports. However
 Sun Microsystems would be interested to receive feedback from customers on
 their experiences with Quagga.

- These binaries were not compiled with vtysh, to avoid having it depend on
 readline. If you have a readline development environment, you can compile
 vtysh from the Quagga sources, using the above configure options, and it
 should work. You can of course use the telnet accessible interfaces.